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NETWORK METRIC SYSTEM 

Field of the Invention 



This invention relates generally to network metric systems and, more particularly, to a 
system and methodology for one-way measurement of network metrics at the Internet 
Protocol layer to produce comparable measurements for network engineering. 

Background of the Invention 

10 

The explosive growth of Internet traffic since the mid-1990s shows no sign of abating 
and may be expected to continue well into the 21 st Century. Accelerating need for bandwidth 
driven by home and business usage has spawned an Internet- working infrastructure managed 
£ by a new industry of Internet service providers (ISPs). As the complexity and connectivity of 
15 the multi-layer Internet communication system grows, expanded usage of audio, voice, and 
1 video across the Internet seems certain to place unprecedented demand on bandwidth 

available now and in the foreseeable future. In this context, it is clear that quality of service 
TC can only increase in importance as a definitive issue for ISPs and their customer base. 

In order to enable control and monitoring capability over Internet services, there is a 
2ft: clear need for precise and scalable metric tools that can give ISPs real-time measurements 

across nodes and groups of nodes, for a variety of packet types. The availability of a practical 
and versatile system capable of real-time measurement of one-way loss, delay, jitter, and 
other parameters defining the quality of service metrics, therefore, would greatly enhance 
layered network functionality and, at the same time, provide a competitive edge for ISPs who 
25 can consistently demonstrate high levels of quality of service performance. 

Others have attempted to gather network measurement data and record benchmarks, 
however, this work has been done almost entirely at the application layer. Measurement data 
generated at the application layer can then only be compared to measurements performed on 
the same or similar applications, as well as on the same platform. Further, these 
30 measurements can only be compared in roughly the same time period so that the application 
versions and operating system versions are also of a comparable type. These prior 
measurement techniques do not produce any basic network layer measurements of the type 
desired by network engineers that are comparable cross application and cross platform. 
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Additionally, other past attempts to gather measurement data over proprietary 
networks and the Internet have utilized a two-way measurement scheme (i.e., a round-trip 
measurement). This measurement technique has many drawbacks which lead to inaccuracies. 
For example, in many network systems, such as the Internet, data packets travel in 
5 asymmetrical paths. This asymmetrical nature creates obvious difficulties in analyzing two- 
way measurement data. For these, and other reasons, two-way measurement schemes are 
very limited in the degree of accuracy that they can provide. 

Another problem with prior data measurement gathering systems is that they have 
been very bandwidth intensive. As these prior measurement techniques use significant 
1 0 bandwidth, the number of measurement points that the system can analyze is limited. Thus, 
once the system has reached only a few dozen measurement points, the system will break 
: fj down due to bandwidth limitations. Moreover, customers are not interested in a measurement 
=~. system that will drastically decrease the efficiency of the network due to the amount of 
■ network traffic produced by the measurement technique. These types of bandwidth intensive 
15} measurement techniques undesirably prevent the measurement system from being scalable 
enough to have functional significance in a real-world network environment. 

Accordingly, those skilled in the art have recognized the need for a system that is 
; ; capable of measuring Internet metrics in a scalable network environment to produce accurate 
3 and comparable measurements. The present invention clearly addresses these and other 
20 needs. 

Summary of the Invention 

Briefly, and in general terms, the present invention resolves the above and other 
problems by providing a network metric system and methodology which provides 

25 comparable measurements over a network at the Internet Protocol (IP) layer for use in 
network engineering and Internet Service Provider (ISP) performance monitoring. The 
network metric system of the present invention utilizes nodal members to form a nodal 
network between which one-way measurements are performed over asymmetrical paths. The 
measurements are performed at the IP layer, and the number of nodal members in the nodal 

30 network is scaleable. More particularly, the nodal members in the network metric system of 
the present invention are used as measurement points and have synchronized timing systems. 
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Preferably, in this regard, the nodal members support Network Time Protocol (NTP) timing 
synchronization and Global Positioning System (GPS) timing synchronization. 

In accordance with one aspect of the present invention, the one-way measurements are 
performed by the nodal members at the IP layer and provide cross-application and cross- 
5 platform comparable measurements. The system utilizes a vector based measurement system 
to achieve service-based, comparable measurements. Preferably, the vector based 
measurement system defines a vector by an IP source, an IP destination, and a service type. 
In accordance with the present invention, the measurements performed between the nodal 
members are selected from a group including, by way of example only, and not by way of 
1 0 limitation, code version, source identities, time parameters, sequence/byte/packet loss, out of 
order packets, error packet types, sequential packet loss, packet hop count, IP protocol 
tracking, packet TOS and DiffServ changes, packet jitter, one-way latency, outages, and route 
information. 

In accordance with another aspect of the present invention, the nodal members of the 
1 5 ; network metric system perform processing of the measurement data. Preferably, the nodal 

members implement a processing algorithm on raw measurement data recorded for each 
- - measurement period. This processing algorithm compacts the raw measurement data. In one 
\: preferred embodiment of the present invention, the raw measurement data is compacted to 
S approximately 1 kilobyte per five minute measurement period per vector. Preferably, the 
20 distributed processing among the nodal members allows centralized processing of the raw 

measurement data to be eliminated. The network metric system minimizes network traffic by 
utilizing the nodal members for distributed processing. Preferably, the network metric system 
eliminates single point failure by utilizing the nodal members for distributed processing. 

In accordance with another aspect of the present invention, the nodal members of the 
25 network metric system are true Internetworking devices, which support TCP/IP, SNMP, 

Telnet, TFTP, dhcp, BootP, RARP, DNS resolver, traceroute, and ping functions. Preferably, 
the nodal members include multiple on-board processors, enabling one processor to handle 
management processes and another processor to handle measurement processes. In one 
preferred embodiment of the network metric system, each nodal member is capable of 
30 automatic software updating in synchronization with other nodal members in the nodal 
network for minimal loss of measurement time and enhanced scalability. 
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In accordance with another aspect of the present invention, the nodal members of the 
network metric system are autonomous devices that are capable of generating measurement 
packets, performing one-way measurements at the IP layer, processing measurement data, 
and temporarily storing measurement data, despite a service daemon or database outage. 
5 Preferably, the nodal members are functional without requiring a TCP session with the 
service daemon. In one preferred embodiment of the network metric system, the nodal 
members employ a dual power system to minimize power failures. In response to a nodal 
member failure, the nodal member preferably records the reason for the failure, and 
automatically reestablishes the nodal member to the nodal network upon resolution of the 
10 failure. 

Another preferred embodiment of the present invention is directed towards a 
measurement method for performing measurements over a network. The method includes: 
performing one-way measurements between nodal members over asymmetrical paths, 
wherein the measurements are performed at the IP layer in a scalable environment; processing 

1 5 data produced by the one-way measurements between nodal members; transmitting the pre- 
processed measurement data from the nodal members to a database; and analyzing the pre- 
processed measurement data. More particularly, the method of performing one-way 
measurements between nodal members is achieved by transmitting measurement packets with 
CQOS headers between nodal members. Preferably, the method of processing the 

20 measurement data produced by the one-way measurements between nodal members also 
compacts the measurement data. 

Another preferred embodiment of the present invention is directed towards a 
measurement system for performing measurements over a network. The system includes a 
nodal network, a database, an application server, a workstation, and at least one service 

25 daemon interfacing with nodal network, and the database. The nodal network includes 
multiple nodal members between which one-way measurements are performed over 
asymmetrical paths. In the network metric system of the present invention, the measurements 
are performed at the IP layer, and the number of nodal members used as measurement points 
in the nodal network is scaleable. The database stores measurement data processed by the 

30 nodal members. The workstation provides a user interface for system configuration, 

including sending vector configuration information to the database, as well as reporting of the 
measurement data. The application server interfaces between the database and the 
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workstation for system configuration and results display (obtaining the results data from 
database and preparing the data for display). The service daemon interfaces with the nodal 
network and the database. Specifically, the service daemon preferably obtains configuration 
information from the database, instructs the nodal members to create vectors (configures the 

5 nodal members), gathers results data from the nodal members, and stores results data 
transmitted from the nodal members to the database. 

In accordance with another aspect of the present invention, the application server of 
the network metric system interfaces with the management/reporting workstation via HTML, 
Java, or CGI for system configuration and results display. Preferably, the service daemon 

1 0 performs automatic error recovery to retrieve missing measurement data when measurement 
data is lost in transmission. In one preferred embodiment of the network metric system, the 
nodal members continue to perform measurements and store measurement data in response to 
a service daemon failure until a replacement service daemon is activated. -■ _ , - 
In accordance with still another aspect of the present invention, the workstation 

15- utilizes a browser based interface to provide system reports and management functions to a 
user from any computer connected to the Internet without requiring specific hardware or 
software. Preferably, the user interface of the workstation is alterable without modifying the 
underlying system architecture. However, the system is capable of performing measurements 
and storing measurement data without dependence upon the user interface. 

20 In accordance with yet another aspect of the present invention, the network metric 

system implements an access protocol that is selectively configurable to allow third party 
applications to access the system. Preferably, the workstation utilizes multiple levels of 
access rights, including, by way of example only, and not by way of limitation, administrator 
level access rights and user level access rights. The administrator level access rights 

25 preferably allow various types of system configuration, including the 

creation/modification/deletion of nodal members, vectors, service types, logical groups of 
vectors, and user access lists, while the user level access rights preferably allow only report 
viewing. 

In accordance with another aspect of the present invention, the network metric system 
30 implements a CQOS protocol, which is a non-processor intensive, non-bandwidth intensive 
protocol for transmitting pre-processed, compacted measurement data. In one preferred 
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embodiment of the network metric system, measurement data from each measurement period 
is sent from the nodal members to the database via this CQOS protocol. The nodal members 
also communicate with each other and obtain results data using CQOS protocol. Moreover, 
configuration data and status data are also sent via CQOS protocol. 

5 In accordance with another aspect of the present invention, the database of the 

network metric system is SQL compliant. In one preferred embodiment of the network 
metric system, the database stores vector configuration information and results of the 
measurement data to allow generation of true averages in response to user defined parameters. 
The data stored in the database preferably includes, by way of example only, and not by way 
1 0 of limitation, code version; nodal member ID; vector ID; measurement period ID; universal 

time; length of measurement period; number of packets and bytes sent and received in the 
*°f measurement sequence; anomalies, including out of order, duplicated, fragmented, dropped, 
«" IP-corrupted, pay load-corrupted, CQOS information corrupted; TTL changes, TOS changes, 
.~ minimum/maximum/average/standard deviation for one-way latency and jitter, and route 
l=fj information. 

? In accordance with still another aspect of the present invention, the one-way 

measurements performed by nodal members at the IP layer provide cross application and 
cross platform comparable measurements. In one preferred embodiment of the present 

p invention, the network metric system utilizes a vector based measurement system to achieve 
2lT service-based, comparable measurements. Preferably, the vector based measurement system 
defines a vector by an IP source, an IP destination, and a service type. The network metric 
system is preferably configured so that vectors in the vector based measurement system are 
capable of disablement without deletion from the database. 

In accordance with yet another aspect of the present invention, the nodal members of 
25 the network metric system implement hardware time stamping. Hardware time stamp is more 
accurate than software time stamping. This system architecture configuration offloads the 
processor-intensive activity of time stamping and frees up processing power. Each nodal 
member includes an output buffer, and during the hardware time stamping, header 
information and data information preferably fill the output buffer before a time stamp is 
30 applied to the output buffer. 
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In accordance with another aspect of the present invention, the network metric system 
provides user-definable groupings of vectors for facilitating vector display and reporting. The 
nodal members in the nodal network are capable of user-defined customizable groupings for 
area-specific measurement reporting. In the network metric system of the present invention, 
5 the customizable groupings of nodal members are capable of overlapping each other. The 
system further preferably allows the measurement reports generated by the system to be 
produced in both standard formats and customized formats. 

In accordance with still another aspect of the present invention, the nodal members of 
the network metric system generate and transmit measurement packets in order to perform 
10 one-way measurements at the IP layer. Specifically, the measurement packets have a format 
that preferably includes an Ethernet header, IP header, optional IP routing options, UDP/TCP 
header, payload, and CQOS header. In a preferred embodiment of the network metric 
— system, checksums are calculated on the measurement packets for payload, IP header, 
UDP/TCP header, and CQOS header. 

1 5 In accordance with yet another aspect of the present invention, the network metric 

system facilitates user-definable bandwidth allocation for measurement traffic. Preferably, 
each nodal member automatically calculates the rate at which measurement packets are 
generated, based upon the number of vectors, packet size, and the bandwidth allocation. In a 
preferred embodiment of the present invention, the network metric system performs accurate 

20 measurements at a high sampling rate. 

Still another preferred method of the present invention is directed towards a 
measurement method that includes: performing one-way measurements between nodal 
members over asymmetrical paths, wherein the measurements are performed at the IP layer in 
a scalable environment; processing data in the nodal members produced by the one-way 

25 measurements between nodal members; transmitting the pre-processed measurement data 

from the nodal members to a database via at least one service daemon that interfaces with the 
nodal network and the database, wherein the at least one service daemon instructs the nodal 
members to create vectors, obtains vector configuration information from the database, and 
processes results data transmitted from the nodal members to the database; and providing for 

30 system management capabilities and measurement data analysis via the workstation. 
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Yet another preferred embodiment of the present invention is directed towards a 
measurement system for performing measurements over a network that also performs a 
readiness test. The system includes a nodal network, a measurement database, a user 
interface workstation, an application server, and a service daemon. The nodal network 
5 includes multiple nodal members between which one-way measurements are performed at the 
IP layer. The workstation provides a user interface for system configuration, including 
sending vector configuration information to the database, as well as reporting of measurement 
data. The application server interfaces between the database and the workstation for system 
configuration and results display (obtaining the results data from database and preparing the 
1 0 data for display). The service daemon interfaces with the nodal network and the database. In 
the network metric system of the present invention, a transmitting nodal member performs a 
readiness test to ensures the willingness of a receiving nodal member to accept measurement 
^1 traffic before the transmitting nodal member begins to transmit measurement traffic to the 
g receiving nodal member. 

1 5 In accordance with the present invention, the readiness test of the network metric 

system preferably includes: broadcasting an Address Resolution Protocol request to a 
gateway/local host in order to obtain its physical hardware address; pinging the gateway/local 
host; pinging the receiving nodal member; performing a traceroute to the receiving nodal 
member; and performing a Go/No Go test using a CQOS protocol which is a non-processor 

20 intensive, non-bandwidth intensive protocol for nodal members to communicate with each 
other. 

In further accordance with the present invention, the Go/No Go test of the network 
metric system is performed by a transmitting nodal member requesting and obtaining 
permission from a receiving device to transmit measurement traffic before the transmitting 
25 nodal member transmits the measurement traffic. This ensures protection against unwanted 
measurements being made on nodal members, as well as against measurement traffic being 
sent to a non-nodal member receiving device. The readiness test verifies linkage and 
reachability of nodal members before measurements are performed without burdening the 
network with unnecessary duplication of effort. 
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Other features and advantages of the present invention will become apparent from the 
following detailed description, taken in conjunction with the accompanying drawings, which 
illustrate by way of example, the features of the present invention. 

Brief Description of the Drawings 

5 

FIG. 1 illustrates a perspective view of the system architecture of the network metric 
system, in accordance with the present invention; 

FIG. 2 illustrates a block diagram of one embodiment of a nodal member used in the 
network metric system of the present invention; 
10 FIG. 3 illustrates an block diagram of another embodiment of a nodal member used in 

the network metric system of the present invention; 

FIG. 4 illustrates a perspective view of a sample report of the network metric system 
of the present invention; and 
-I FIG. 5 illustrates a perspective view of an alarm screen of the network metric system 

1 5 of the present invention. 

7 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

A preferred embodiment network metric system and methodology, constructed in 
accordance with the present invention, provides comparable measurements over a network at 
the Internet Protocol (IP) layer for use in network engineering and Internet Service Provider 

20 (ISP) performance monitoring. The network metric system is capable of measuring one-way 
Internet metrics in a scalable network environment to produce accurate, comparable 
measurements. Referring now to the drawings, wherein like reference numerals denote like 
or corresponding parts throughout the drawings and, more particularly to FIGS. 1-5, there is 
shown one embodiment of a network metric system 10 constructed in accordance with the 

25 present invention. 

Referring now to FIGURE 1, the network metric system 10 includes a nodal 
network 20, a database 40, an application server 46, a workstation 50, and at least one service 
daemon 60 that interfaces between the workstation 50, the nodal network 20, and the database 
40. The nodal network 20 is composed of a plurality of nodal members 30 between which 
30 one-way measurements are performed over asymmetrical paths. In the network metric 
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system 10 of the present invention, the measurements are performed at the IP layer, in 
contrast to prior systems that performed measurements at the application layer. Further, the 
number of nodal members 30 used as measurement points in the nodal network 20 is highly 
scalable, in order to allow accurate measurements to be performed in network environments 
5 of virtually any size. The database 40 stores measurement data that is generated by the nodal 
members 30. The workstation 50 is connected to the database 40 via the application 
server 46, and provides a user interface for system configuration, including sending vector 
configuration information to the database. The workstation 50 also provides a user interface 
for reporting of the measurement data. The application server 46 interfaces between the 
1 0 database 40 and the workstation 50 for system configuration and results display. Results 
display includes obtaining the results data from database 40 and preparing the data for 
display. One or more service daemons 60 interface between the nodal network 20 the 
^ database 40. 

In a preferred embodiment of the network metric system 1 0, measurements are 
1 5i accomplished by transmitting CQOS measurement packets from one nodal member 30 to 
another nodal member 30. This measurement is made of a one way trip, which is a major 
I improvement over traditional methods (using ping or similar techniques) that measure round 
--. trip times. In general, these measurements are made with nanosecond resolution when a 
'J2 Global Positioning System (GPS) time synchronization system is utilized. These 
20 measurements are made with up to 1 millisecond resolution when using a network time 
protocol (NTP) synchronization system. In one preferred embodiment of the present 
invention, results are calculated based upon a 5 minute measurement period and are 
transmitted from the receiving nodal member 30 to the database 40 for later analysis. 

In a preferred embodiment of the present invention, a vector is used to describe a 
25 measurement case. Each vector has a start point and an end point. The start point is the 
nodal member 30 that is transmitting CQOS measurement packets to the receiving nodal 
member 30, the later of which is the end point. Hereinafter, the terms transmitter and receiver 
are considered equivalent to start point and end point nodal members, respectively. 

In a preferred embodiment of the network metric system 1 0 of the present invention, a 
30 vector is the fundamental definition of the path and measurement traffic between two nodal 
members 30 for the calculation of measurements at the IP layer. As the fundamental 
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measurement service element, a vector describes the path and measurement traffic type 
between two nodal members 30. It is uniquely defined by a measurement packet between 
specific IP source and destination addresses. In this embodiment, a vector is defined by an IP 
source, an IP destination address, and a service type (differentiated service bits), with user- 
5 definable TCP/UDP ports, payload (zeros, ones, or random), and packet size. This 

fundamental IP layer metric allows for service-based, comparable measurements that translate 
cross-application and cross-platform. With this flexibility, customers can configure vectors 
to create high-fidelity measurements that exactly match their existing and/or planned IP 
traffic. 

1 0 Each vector has an associated set of characteristics. These characteristics include 

items such as packet size, payload type, header type (none/UDP/TCP), udp/tcp source and 
destination port numbers, TOS/DiffSev bits, TTL value, IP protocol value, IP options, default 
gateway, source and destination addresses, and TCP header information. Further, a certain 
: set of characteristics can be assigned a name such as 'high priority' or 'best effort'. This 

15 makes it easy to reuse a particular set of characteristics. 

In a preferred embodiment of the network metric system 10, all measurements are 
~ r made on the end point nodal member 30. The nodal member 30 is called the vector handler. 
It is the responsibility of the transmitter to send out measurement packets to the receiver. It is 
also the responsibility of the transmitter to send out an ending packet at the end of each 

20 measurement period. This ending packet signals the receiver that all packets in the 

measurement period have been transmitted. Once the receiver acquires the ending packet at 
the end of the measurement period, the receiver becomes responsible for gathering the data of 
all packets received from the transmitter, calculating the results based on the data contained in 
the packets, and finally sending the results to the database 40 for storage. 

25 In a preferred embodiment of the network metric system 1 0 of the present invention, 

the CQOS service daemon 60 is the foundation of the scalable and reliable application server 
architecture. In a preferred embodiment, the service daemon 60 interfaces with the nodal 
members 30 and the database 40, instructs the nodal members to create new vectors, obtains 
vector configuration information from the database 40, and handles results data transmitted 

30 from the nodal members 30 to the database 40. Initially, vector configuration information is 
sent from the workstation 50 through the application server 46 to the database 40. In some 
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embodiments to the present invention, multiple service daemons 60 are run simultaneously to 
provide for system redundancy. In the embodiments of the present invention that utilize 
multiple service daemons 60, the system 10 employs a Solaris based operating system, 
instead of Windows NT. If a service daemon 60 experiences a failure, the nodal members 30 
5 continue to measure and store their results until a replacement daemon is activated. 

In a preferred embodiment of the present invention, the service daemon 60 allows the 
network metric system 10 to be self-sustaining, with measurements performed, and results 
stored, without dependence upon the user interface. Further, the service daemon 60 allows 
the user interface to be changed or otherwise updated without affecting the underlying system 
1 0 architecture. Moreover, the service daemon 60 preferably allows the flexibility to potentially 

let third-party applications access the measurement system 10, as desired. 
3 In a preferred embodiment of the network metric system 10 of the present invention, 

the one-way measurements are performed by the nodal numbers 30 and provide cross- . 
^ application and cross-platform comparable measurements. As described above, the system 
15 utilizes a vector-based measurement system 10 to achieve service-based, comparable 

measurements between the nodal numbers 30. Specifically, the vector-based measurement 
= system 10 defines a vector using an IP source, an IP destination, and a service type. 
'£ A nodal member 30 can be configured to be the start point or end point of many 

"r vectors simultaneously. Note that the packet sent out at the end of each measurement period 
20 is not sent for each vector, but rather it is sent on a per nodal member 30 basis. For example, 
if one nodal member 30 is the transmitter of two vectors to the same receiving nodal member, 
the transmitting nodal member only sends one packet at the end of the measurement period, 
not two. 

The nodal members 30 in the network metric system 10 of the present invention 
25 perform measurements and store measurement data over a set measurement period. As 
described above, the results are preferably calculated based on a 5 minute measurement 
period. However, any desired measurement period may be used in other embodiments of the 
present invention. The results data for each measurement period is sent from each nodal 
member 30 to the database 40 utilizing the CQOS protocol for later analysis. The CQOS 
30 Protocol is a communications protocol that is used for communication between nodal 

members 30 and the other elements of the network metric system 10. The results for each 
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measurement period are sent from each nodal member 30 to the service daemon(s) 60 and 
then onwards to the database 40 utilizing the CQOS protocol. Moreover, configuration data 
and status data are also sent via CQOS protocol. 

The CQOS protocol is an efficient, secure, non-processor intensive, non-bandwidth 
intensive transfer protocol. Use of the CQOS protocol allows processor and bandwidth 
intensive protocols such as Simple Network Management Protocol (SNMP) to be avoided. 
The CQOS protocol is also used for communication between nodal members 30. Moreover, 
the CQOS protocol can be expanded and modified, as needed, throughout the development 
life cycle of the product. 

Set of Metrics 

The network metric system 10 of the present invention measures and reports a 
complete set of Internet metrics that are useful to network engineers for proper network 
design and configuration. The completeness of these Internet metrics provides significant . 
advantages over prior measurement gathering systems. Specifically, the Internet metrics, in 
accordance with the present invention, preferably include, by way of example only, and not 
by way of limitation, code version number, source identities, time parameters, 
sequence/byte/packet loss, out-of-order packets, air packets' types, sequential packet loss 
(loss patterns), packet hop count, IP protocol tracking, packet TOS and DiffServ changes, 
packet jitter, one-way latency, outages, and route information. Furthermore, many of these 
Internet metrics can be subdivided and described in further detail. 

The code version number provides the version number of software operating in the 
nodal members 30, which is important when updates are made or are being planned. In 
source identities, the sending nodal member ID should be recorded as well as the sending 
vector ID. Regarding the sending nodal member ID, all nodal members 30 have a hard-coded 
identity and can be named. With respect to the sending vector ID, a default identifier of all 
vectors is automatically created. 

In the time parameter category, specific metrics include measurement period ID, nodal 
measurement period ID, and universal time. The measurement period ID is defined as 
continuous time divided into periods identified by measurement ID. The nodal member 
measurement period ID relates to the measurement period of the nodal member that is 
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transmitting packets. The universal time metric provides an absolute time reference for all 
measurements. 

Several Internet metrics relate to sequence, byte, and packet loss. These include 
sequences received, bytes received, bytes transmitted, packets received, and packets 
5 transmitted. Referring to the sequences received metric, when packets are sent to multiple 
nodal members 30, each nodal member receives a sequence of packets in turn. The number 
of sequences received is counted separately from the number of bytes and packets received. 
In order to measure sequential packet loss (the number of packets dropped in a row), it is 
necessary to be able to identify the sequence in which the packet was sent. This should be 
1 0 indicated per measurement period. Packet loss is calculated as the number of packets 
transmitted minus the number of packets received. Packet loss does not take account of 
□ duplicate packets. The bytes received metric refers to the number of bytes received per 
*~ measurement period. Bytes transmitted is defined as the number of bytes transmitted per 

- each measurement period. Packets received is defined as the number of packets received per 
1-5= measurement period. Finally, packets transmitted is defined as the number of packets 

transmitted per measurement period. The out-of-order packets metrics category includes a 

- = measurement for packets out of order and groups out of order. Referring to the packets out of 

order measurement, nodal members 30 implement the sophisticated algorithm described 
"~\ above to calculate the number of packets that arrive out of order. Since such packets may be 
2tf grouped together, the system also applies the algorithm to groups of out-of-order packets to 
produce the group's out-of-order measurement. 

Error packet types are a large category of Internet metrics. These include packets 
duplicated, minimum packets duplicated, maximum packets duplicated, packets dropped, 
packets dropped due to missing fragment, packets fragmented, minimum packets fragmented, 
25 maximum packets fragmented, average packets fragmented, IP packets corrupted, CQOS info 
packets corrupted, pay load packets corrupted, and optional header packets corrupted. The 
packets duplicated metric is produced by identifying duplicated packets and accounting for 
duplicated packets in the calculation of packet loss. The packets dropped metric identifies the 
packets transmitted and the number of which were dropped. This calculation takes account of 
30 duplicated packets. The packets dropped due to the missing fragment metric accounts for 
packets that were received but counted as drop packets due to missing fragments. The 
packets fragmented metric is defined as the number of packets received that were fragmented. 
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In the IP packets corrupted metric, the nodal members 30 identify corruption in the IP header. 
In the CQOS information packets corrupted metric, the nodal member 30 identifies 
corruption in the CQOS information field. In the payload packets corrupted metric, the nodal 
member 30 identifies corruption in the payload. Finally, in the optional header packet 
5 corrupted metric, the nodal member 30 identifies corruption in the optional header. 

The sequential packet loss (loss patterns) category also preferably includes numerous 
sub-categories of desirable metrics. These include minimum sequential packets dropped, 
maximum sequential packets dropped, average sequential packets dropped, standard deviation 
of sequential packets dropped, minimum sequential packets lost, maximum sequential 
10 packets lost, average sequential packets lost, and standard deviation of sequential packets 
lost. All of these sequential packet loss pattern metrics are calculated using the number of 
"2 packets dropped in immediate succession to each other. These calculations are performed for 
~ both lost and duplicated packets. 

The packet hop count category of metrics preferably includes the sub-categories of 
1% packets TTL changes, packets TTL minimum, packets TTL maximum, and packets TTL 
average. For each of these packets TTL-based metrics, the measurements are calculated by 
using the hop count derived from the changes in the time-to-live field in the IP header of the 
packet. TTL (time to live) is a function that limits the life of a packet to a designated number 
of hops between nodal members 30. The time-to-live function is useful in identifying the 
20 length of a path taken by a packet between two nodal members 30, and is particularly useful 
with respect to packets that move along asymmetrical paths. 

In the network metric system 10 of the present invention, the Internet metrics being 
recorded also include packet IP protocol errors and packet IP protocol changes within the 
category of IP protocol tracking. Further Internet metrics being tracked include the category 
25 of packet type of service (TOS) and differentiated services (DiffServ) changes. Subcategories 
of metrics within the packet TOS and DiffServ changes category include the packets TOS 
changes metric, in which the nodal members 30 record differences in the TOS field, as well 
as the packets first ten TOS count metric. 

Still another Internet metrics category is packet jitter. Further metrics within this 
30 category include jitter minimum, jitter maximum, jitter average, jitter standard deviation, and 
jitter standard deviation power 4. The jitter standard deviation power 4 metric allows 
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calculation of statistical accuracy from which minimum, maximum, and standard deviation 
for jitter are reported. 

One-way latency is another general category of metrics under which several specific 
Internet metrics are preferably tracked. These include latency minimum, latency maximum, 
5 latency average, latency standard deviation, latency standard deviation power, and latency 
time stamp mismatch. The latency standard deviation power metric is used to allow 
calculation of statistical accuracy, from which the minimum, maximum, and standard 
deviation for jitter are reported. 

Another Internet metric's category of outages in the network metric system 10 of the 
1 0 present invention includes the subcategories of outages, outage duration, minimum outages, 
outage duration maximum outages, and outage duration total outages. These subcategories of 
outage metrics are calculated by using a certain period measured in nanoseconds after which 
T- an outage counter is started if no packets are received. The outage counter is stopped when 
h the first new packet is received. 
1 ■£; The final category of Internet metrics that is tracked by a preferred embodiment of the 

network metric system 10 is that of route information. The system records first and last 
; packet information for all packets of a measurement period that have IP options set for record 
£ route, strict route, or loose routes. The record route function records the actual path taken by 
(X a packet between two nodal members 30. The strict route function forces a packet to take a 
20 specific path of travel between two nodal members 30. The loose route function allows the 
packet to take any path as it is routed between nodal members 30. The specific sub- 
categories of Internet metrics recorded within the route information category include first 
route type, first route count, first route packet ID, first route data, last route type, last route 
count, last route packet ID, and last route data. 

25 VectorHandler 

The VectorHandler class is used to encapsulate all received packets and result 
calculations for a single measurement period. It inherits from the AtomicAlgorithms that 
contains all of the result calculation routines except for one, the CalculateResults routine. 

CalculateResults Method 
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This method is called two minutes after the measurement period is over and the 
ending packet, indicating that all packets have been sent, arrives from the transmitter. This 
method retrieves the packets for a given measurement period. It then retrieves the non-unique 
0 based period ID from the first packet with a non-corrupted CQOS header. After allocating 
5 the required memory to calculate the results, it calls additional methods to do most of the 
calculations (specifically the methods listed in the AtomicAlgorithms section). This method 
then gathers the version information, temperature information, vector identification 
information, additional vector information, route information, and port counters and places 
them in the results structure. Finally, it calls a method to place the results into the hash tables 
1 0 for temporary storage before transmitting them out to the database on another computer. 

Inputs: nodal memberMPeriodID - unique period ID on which the measurement 
period calculates results. 

Outputs: None 

~ Returns: TRUE if results were calculated, FALSE if results could not be calculated. 

15" AtomicAlgorithms 

™ This class contains all of the methods that are used by VectorHandler, which inherits 

this class, to calculate results from the AtomicPacketData linked lists for a measurement 
period. 

mcProcessFirstPass Method 

20 This method loops through all of the AtomicPacketData packets and places all packets 

with non-corrupted CQOS headers in the rlnfo array. During this process, the method saves 
any information in the results struct that can be obtained from the packets even if certain 
headers are corrupted or duplicates occur. The main results it calculates are bytes received, 
packets received, fragmentation, TTL, IP protocol, TOS, latency, and header error results. 

25 Inputs: results - pointer to results struct 

atomic - pointer to head of AtomicPacketData linked list with all packets 

rlnfo - point to preallocated array to hold all packets with non-corrupted 

CQOS headers 

rCount - maximum number of items rlnfo array can hold 
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Outputs: results - saves appropriate results calculated in this method 

rlnfo - array with all packets with non-corrupted CQOS headers 
Returns: all OxF's if error, number of valid items in rlnfo array if no errors 
mc ProcessSecondPass Method 

This method allocates memory for duplicated packets and sorting arrays. It then loops 
through all of the packets placed by mc_ProcessFirstPass into the rlnfo array and places all 
duplicates found into the duplicated packets array. Next, the method sorts the items in the 
rlnfo array which are not duplicates into transmission order. Based on the sorted order it 
calculates jitter by looping through the packets in order transmitted. Outage results are then 
computed by looping through the packets in the order received and comparing the times 
received with the min outage value ignoring duplicates. Duplicate results are then calculated 
by looping through the items previously placed in the duplicate list. Finally, all values 
previously calculated are placed in the results structure. 

Inputs: results - pointer to results struct 

rlnfo - array with all packets with non-corrupted CQOS headers 

rCount - count of all packets received 
txPackets - count of packets transmitted 
rxPackets - count of all packets received 

outageTriggerTimeNS - minimum time considered for an outage in nsec 
mPeriodNanoseconds - UTC time of period in nsec 

nodal memberVerifyRxTimestamp - time end of period message was received 

outageCoolCount - number of items to check to be able to verify an outage without 
using end of period time 

Outputs: results - saves appropriate results calculated in this method 

rlnfo - array with all packets with non-corrupted CQOS headers with duplicate 
information added by this method 

Returns: all OxF's if error, number of duplicated items if successful 
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mc ProcessThirdPass Method 

This method loops through an array of rlnfo items to find the number of items, and 
groups of items that are out of order. It places those results in the rxGroupsOutOfOrder and 
rxPacketsOutOfOrder result fields. 
5 Inputs: results - pointer to results struct 

rlnfo - point to preallocated array to hold all packets with non-corrupted 

CQOS headers 

rCount - number of packets received 
Outputs: results - saves appropriate results calculated in this method 
lft Returns: FALSE if an error occurs, TRUE if successful 

.== mc IsDunlicatedAbove Method 

This method used by mc_ProcessSecondPass to loop through duplicates that are 
Z before the current item. If a duplicate is found before the item then TRUE is returned. 
Otherwise FALSE is returned. 

IS Inputs: r - pointer to current item 

j currentPosition - index of current item in r array 

C\ duplicatedList - array of indexes of duplicated items 

duplicatedListCount - number of items in duplicated list 
rlnfo - array of PACKETRecordlnfo items 
20 rCount - number of items in rlnfo array 

Outputs: None 

Returns: TRUE if duplicated above, FALSE if not 
mc FindTransmittedPacket Method 

This method is used by mc_ProcessSecondPass to loop through an array of rlnfo 
25 items to try and find a packet with the correct sequence number. 

Inputs: sequence - number of sequence to look for 
rlnfo - array of PACKETRecordlnfo items 
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rCount - number of items in rlnfo array 
startlndex - index of where item should be 
Outputs: None 

Returns: all OxF's indicates not found, index number of packet 
mc StripBee Method 

This method is used by mc_ProcessThirdPass to find groups at the beginning of the 
array and return the new first index of the array. It also updates the current minimum value. 

Inputs: rlnfo - point to preallocated array to hold all packets with non-corrupted 
CQOS headers 

FirstPosition - index to start from 

LastPosition - max index to stop at 

Marked - array of indexes already marked 

CurrentMin - the current minimum value 
Outputs: CurrentMin - the new min (could stay the same) 
Returns: The new first position 
mc StripEnd Method 

This method is used by mc_ProcessThirdPass to find groups at the end of the array 
and return the new last index of the array. It also updates the current maximum value. 

Inputs: rlnfo - point to pre-allocated array to hold all packets with non-corrupted 
CQOS headers 

FirstPosition - index to start from 

LastPosition - max index to stop at 

Marked - array of indexes already marked 

CurrentMax - the current maximum value 
Outputs: CurrentMax - the new max (could stay the same) 
Returns: The new last position 
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Packet Format 

The measurement packets, in accordance with a preferred embodiment to the present 
invention, utilize a specific, efficient packet format. This packet format includes all of the 
pertinent information required for the methodology of the network metric system 10 of the 
present invention. In one embodiment of the present invention, the packet format is 
configured as: Ethernet header, IP header, optional IP options (strict, loose, or record route), 
TCP/UDP header, payload, and CQOS data. Preferably, check sums are calculated for 
payload, IP header, TCP/UDP header, and CQOS header. 

COOS Measurement Packet Structure 

Shown below is one format of a CQOS measurement packet. It consists of an 
Ethernet Header and checksum, an IP header, the payload, and a CQOS header. These items 
are briefly described in the sections that follow except for TCP/UDP headers. TCP/UDP 
headers are not discussed because measurement of TCP/UDP packets to application ports is 
not measured. 



Ethernet Header (14 bytes) 
IP Header (20 - 80 bytes) 
Payload (46 - 1500 bytes with IP, TCP/UDP, and CQOS header) 
CQOS Header (88 bytes) 
Ethernet Checksum (4 bytes) 



Ethernet Protocol and Header Information 

The Ethernet protocol is the protocol actually used to physically transport packets to 
and from the nodal members 30, and to and from the router connected to the nodal members. 
3 0 The format of an Ethernet packet is shown below. 



Ethernet destination address (first 32 bits) 
Ethernet dest (last 16 bits) | Ethernet source (first 16 bits) 
Ethernet source address (last 32 bits) 
Type code (16 bits) 1 

Payload (368 - 12000 bits) 
Ethernet Checksum (32 bits) 
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The Ethernet destination address is a 48 bit unique identifier of the Ethernet controller 
to receive the packet. The Ethernet source address is a 48 bit unique identifier of the Ethernet 
controller transmitting the packet. The payload is the portion where TCP/UDP, IP and CQOS 
5 header information resides. It also is the portion where any other data sent is contained. The 
maximum size of the payload section is 12000 bits which defines the maximum size of data 
that can be sent per packet. The Ethernet checksum is a 32-bit value that is used to validate 
the contents of the entire Ethernet packet. 

IP Protocol and Header Information 
1 0 The IP protocol is used to transport packets across the Internet regardless of the actual 

connection protocols between routers. This protocol lies at the heart of the Internet and its 
-■j header fields contain information that is saved in the results. 

" ; ; Bits |0 3| 7|8 15 116 31 
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] Version | IHL 


IType of Service | 


Total Length | 




| Identi 


fication I Flags | 


Fragment Offset | 


20 


| Time to Live 


| Protocol I 


Header Checksum | 


1 


Source Address 






1 


Destination Address 





The version field contains the current version of IP (normally 4). The IHL field 
contains the length of the header in 32 bit words. This is normally 5 except when an IP 
optional header is used in which case it can be up to 15. (Verify IP optional header size.). 
The Type of Service (TOS) field contains priority information that may or may not be used 

30 by routers to give packets higher or lower priority. The Total Length field specifies the total 
length of the packet (excluding the Ethernet header and checksum) in bytes. The 
Identification field is used to identify the packet. 

The Flags field (3 bits) is used in fragmentation. The first bit, if set, signifies that 
routers should not fragment the packet. If a router must fragment a packet and the first bit is 

35 set, the router will drop the packet. The last bit, if set, signifies that there are more packets 
after this packet that were originally part of one packet but were fragmented into smaller 
ones. The Fragment Offset (13 bits) is the offset from the previous beginning of the original 
packet if it is fragmented into smaller pieces. It is in units of 8 bytes. The Time to Live 
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(TTL) field indicates the max number of hops that this packet can take before reaching the 
receiver or the packet is dropped. The Protocol field indicates the transport protocol used 
(ICMP = 1, IGMP = 2, TCP - 6, UDP = 17). 

The Header Checksum is used to validate the contents of the IP header. To calculate 
5 the checksum, all fields in the IP header (except for this field that is ignored) are treated as 
16-bit numbers and complemented. Then all are summed and stored here. Upon receiving 
the packet all are summed and if all l's then the header is not considered corrupt. The Source 
Address contains the IP address of the transmitting host. The Destination Address contains 
the IP address of the receiving host. 
10 COOS Protocol and Header Information 

The CQOS header is contained at the end of the Ethernet payload. This header 
~~ contains original values of data that can be changed during transmission of a packet. It is 
located by subtracting the size of the CQOS header (88 bytes) from the end of the payload 
section. If the packet is corrupted, CQOS header can also be found because the first field is 
1 5 64-bit ASCII field that contains CQOS. 

I Taglnfo (first 32 bits) I 

| Taglnfo (last 32 bits) I 

2& 

I Version I Reserved 0 (first 16 bits) | 

| Reserved 0 (last 16 bits) I Resevedl I TOS | 

25 | TTL | IP Protocol I Payload Checksum (first 16 bits) | 

I Payload Checksum (last 16 bits)|Header Checksum (first 16 bits) | 
IHeader Checksum (last 16 bits)|nodal member ID (first 16 bits) | 

30 

I nodal member ID (next 32 bits) I 

Inodal member ID (last 16 bits) | nodal Period ID (first 16 bits) | 



35 | nodal member Period ID (next 32 bits) I 

Inodal Period ID (last 16 bits) | Vector ID (first 16 bits) | 

| Vector ID (next 32 bits) I 

40 

I Vector ID (last 16 bits) | Period ID (first 16 bits) | 

| Period ID (next 32 bits) I 

45 | Period ID (last 16 bits) | Burst ID (first 16 bits) | 

| Burst ID (next 32 bits) I 
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| Burst ID (last 16 bits) I Packet ID (first 16 bits) | 

| Packet ID (next 32 bits) I 

5 | Packet ID (last 16 bits) I Tx Timestamp (first 16 bits) I 

| Tx Timestamp (next 32 bits) I 

| Tx Timestamp (last 16 bits) | Not Tx Timestamp (first 16 bits) | 

io 

I Not Tx Timestamp (next 32 bits) I 

| Not Tx Timestamp (last 16 bits) | Not Used I 

15 

The Taglnfo field contains the identifier of the beginning of the CQOS which consists 
of the ASCII CQOS value and is used to find the header if the parts of the packet are 
corrupted. The Version field contains the version of the protocol -1 . The TOS field contains 
the original TOS set on the transmitting nodal member 30. The TTL field contains the 
2Et original TTL set on the transmitting nodal member 30. The IP protocol field contains the 
J original IP protocol set on the transmitting nodal member 30. The P Checksum (Payload 
=i Checksum) field contains a checksum for the entire payload. The H Checksum (Header 

Checksum) field contains a checksum for the CQOS header, 
r The nodal member ID field contains the unique ID of the transmitting nodal member 

25 30. The nodal member Period ID field contains the unique ID of the period for the nodal 
^ member 30. The Vector ID contains the ID of the vector. The Period ID contains the 0 based 
ID of the measurement period. The Burst ID contains the identifier of the burst that this 
packet is in. The Packet ID contains the identifier of this packet (sequence number). The Tx 
Timestamp contains the timestamp of the packet when it was transmitted. The Not TX 
30 Timestamp field contains the inverse of the Tx Timestamp field so that the field can be 
verified even if other parts of the header is corrupted. 
Nodal Member Hardware 

Referring now to FIGURES 2 and 3, in one preferred embodiment to the present 
invention, the nodal members 30 contain on-board intelligence, multiple on-board processors, 
35 64-bit counters, full Internet-working functionality, Ethernet ports, a rack-mountable 
configuration, dual modes of type synchronization, and intelligent upgrading. In one 
embodiment, each nodal member 30 has two 10/100 MBPS Ethernet ports. Preferably, one 
port is used for measurement traffic and in-band management traffic. The second port may 
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optionally be used for out-of-band management. This configuration provides the benefit of 
allowing management traffic to be run on a separate management network. 

In the network metric system 10 of the present invention, the nodal members 30 are 
designed with feature expansion in mind, and with room for additional measurement network 
interfaces. In a preferred embodiment of the present invention, the nodal members 30 are 
rack-mountable devices that include two U-boxes with front panel LEDs, an IrDA port, and a 
serial port. Preferably, a command line interface is also accessible through either the serial 
port, IrDA port, or Telnet. This rack-mountable configuration provides desirable space 
efficiency. Further, the IrDA port eliminates the requirement for a serial cable for basic 
configuration and diagnostics. This also allows CE devices and palm pilot devices to be used 
for configuration. 

There are two main components that comprise the nodal members 30, Component 1 
(a.La. Big Joe), and Component 2 (a.k.a. Mercury). Each component is responsible for 
different tasks and has different connected interfaces. Component 1 contains the time 
stamping hardware, an Ethernet controller, and a microprocessor. It connects to the auxiliary 
serial port at the back of the box, the GPS connector, the PPS signal, the Ethernet 
Measurement port, and Component 2. Component l's main responsibility is to transmit and 
receive packets. When transmitting or receiving packets, Component 1 places a very accurate 
time stamp in the packet (as described below). Packets received are sent to Component 2 for 
further processing. 

Component 2 contains an Ethernet controller and a microprocessor. It connects to the 
serial port at the front of the box, the PPS signal, the IRDA interface, the Ethernet Auxiliary 
port, and Component 1 . Component 2's responsibility is to keep track and store vectors and 
their respective packets, calculate results at the end of measurement periods, and handle any 
high level protocols. The results previously mentioned are calculated on Component 2, 
except for the layer 2 calculations. All the classes and methods described below are 
contained in Component 2. 

Time Stamping 

In one preferred embodiment of the present invention, the nodal members implement 
hardware time stamping. Hardware time stamp is more accurate than software time stamping. 
Additionally, the hardware time stamping offloads the processor-intensive activity of time 
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stamping to free up processing power. Preferably, the time stamp is applied to the output 
buffer after the header information and data information fill the output buffer, so as to more 
closely represent the time at which the measurement packet is actually transmitted. Using 
this technique, the time stamp is generated very close to the actual transmit time, such that 
any remaining delay between the time request and the application of the time stamp, or the 
transmission of the packet, is discernable with substantial accuracy to permit advancing the 
time stamp to actual transmission time. As a result, the latency time, as measured by 
receiving input to the receive nodal member 30, is substantially devoid of inaccuracy due to 
processing times and processing variations in the transmitting nodal member 30. 

Because the time stamp is generated a short period before it is applied to the packet 
and the packet is output, the delay between generation of the time stamp and application or 
packet output, is predictable with substantial accuracy. Unlike conventional systems, the 
time stamp is not generated before the output buffer begins to fill, and therefore, is not subject 
to processing delays and irregularities that precede filling the output buffer. Consequently, 
the time stamp generated can be advanced by a predictable time increment such that the time 
stamp actually correlates to the time at which the time stamp is applied to the packet, or when 
the packet is output to the ISP transmission path. This allows application of a time stamp that 
is initiated at the time at which the packet is formed, or transmitted, not an earlier time. 

In a preferred embodiment of the network metric system 10, the receiving nodal 
member 30 similarly generates a time stamp as the packet fills the input buffer, rather than 
after the packet is further processed. As such, the receive time stamp is offsetable by a 
predictable time delay to correlate to the time at which the packet is actually received at the 
receiving nodal member 30. One-way signal latency may, therefore, be accurately 
determined with a minimum of corruption due to variable internal processing within the 
sending and receiving nodal members 30. 
Node Processing 

In a preferred embodiment of the network metric system 10 of the present invention, 
each nodal member 30 includes sufficient onboard intelligence to perform processing of the 
measurement data for each measurement period. This is achieved by implementing a 
complex algorithm (described in detail below) and compacting the results, preferably to one 
kilobyte per five minute measurement period per vector. This distribution of intelligence to 
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each nodal member 30 allows the system to eliminate centralized processing of the raw data. 
Further, this onboard intelligence and processing ability of the nodal members 30 minimizes 
the results traffic on the network, thus, increasing scalability as a result of this distributed 
processing. Moreover, this system architecture eliminates the problem of single-point failure. 

5 Each nodal member 30 stores up to 48 hours of vector information in a circular buffer. If the 
receiving nodal member 30 does not receive a packet signaling the end of a vectors 
measurement period within that period, the vector information for that period is considered 
invalid and is discarded. 

A preferred embodiment nodal member 30 of the network metric system utilized 
1 0 multiple on-board processors. This allows one processor to handle management processes, 
while another processor handles measurement processes. This configuration also has the 
benefit of increasing scalability of the system. Further, the nodal members 30 in one 
preferred embodiment of the present invention utilize counters with exclusively 64-bit values. 

J: This allows wrapping of the counters to be avoided. 

15 In a preferred embodiment network metric system 1 0, the nodal members 30 are true 

Internet working devices, which are capable of supporting TCP/IP, SNMP, Telnet, TFTP, 
~ dhcp, BootP, RARP, DNS Resolver, Trace Route, and PING. The nodal members 30 are 
high-quality devices that service providers can confidently deploy and manage within their 
own systems. 

20 The nodal members 30 in the network metric system 1 0 of the present invention have 

synchronized timing systems. In this regard, the nodal members 30 preferably support 
network time protocol (NTP), Version 3. A preferred embodiment of the present invention 
supports synchronization to multiple NTP servers. This synchronization is used in the 
calculation of one-way latency and jitter measurements. The one-way latency measurements 

25 provide insight into the asymmetric behavior of networks, and adds a dimension of 

understanding of the performance of real-time applications (voice and multimedia). A 
preferred embodiment of the present invention also supports global positioning system (GPS) 
time synchronization, however, the system 10 avoids dependence solely on GPS which can 
sometimes be difficult to support. 

30 Advantageously, nodal members 30 of the present invention are preferably capable of 

intelligent upgrading. In this regard, the upgrading of the nodal members 30 is automated, 
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and as such, facilitates extreme scalability up to very large numbers of deployed nodal 
members 30, while maintaining minimal loss of measurement time. This ability greatly 
enhances ease of upgrading large deployments. Moreover, after download, new images are 
booted on all nodal members 30 in a synchronized fashion. 

In a preferred embodiment of the network metric system 10 constructed in accordance 
with the present invention, the system implements several redundant features in order to 
account for any occasional failures or errors in the system. The nodal members 30 are 
equipped with a substantial amount of memory storage capacity (typically as RAM) and store 
results data for a period of time after the results have been sent to the database 40. If a results 
packet is lost in the transmission, the service daemon 60 senses this loss and implements the 
necessary procedures to retrieve the results. This type of automated error recovery allows for 
the network metric system 10 of the present invention to act as a carrier class, long-term, 
unattended system deployment. 

In one preferred embodiment of the network metric system 10, each nodal member 30 
employs dual power supplies in order to provide for a backup power source in the case of a 
power supply failure. Moreover, in accordance with the autonomous nature of the nodal 
numbers, if a transmitting nodal member 30 is restarted for any reason, the nodal member 30 
automatically goes through a Readiness Test and a Go/No-Go Test (described below), 
followed by the automatic resumption of measurements without any required user 
intervention. Correspondingly, if a receiving nodal member 30 is restarted and loses its 
vector handlers, then the nodal member 30 automatically sends a message back to the 
transmitting nodal member 30 indicating that the receiving nodal member 30 does not have a 
vector handler for the packets that the transmitting nodal member 30 is sending. The 
transmitting nodal member 30 then goes through its tests, and normal operation is resumed. 
Advantageously, during such temporary outages as described above, the time periods for 
which there is no data are correctly accounted for as downtime for a nodal member 30, and 
not lost measurement packets. 
Readiness Test 

As described above, in a preferred embodiment of the present invention, each 
transmitting nodal member 30 insures the readiness of a receiving nodal member 30 before 
the transmitting nodal member 30 begins to send measurement traffic to another receiving 
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nodal member 30 by performing a Readiness Test. This Readiness Test verifies linkage and 
reachability between nodal members 30 before a test is run, without overburdening the 
network with unnecessary duplication of effort. Specifically, in a preferred embodiment 
network metric system 10, a transmitting nodal member 30 performs a five-step Readiness 
Test upon creation of a new vector by the service daemon 60, or after a restart or other 
anomaly. These steps include: (1) broadcasting and address resolution protocol request to a 
gateway/local host in order to obtain its physical address; (2) pinging the gateway/local host; 
(3) pinging the destination nodal member 30; (4) performing a trace route to the destination 
nodal member 30; and (5) performing a Go/No-Go Test using the CQOS protocol. 

In a preferred embodiment of the present invention, the Go/No-Go Test provides 
protection from unwanted or unauthorized measurements being made on nodal members 30 
within the system, as well as providing protection from having nodal member 30 
measurement traffic accidentally sent to a non-nodal member device. Additionally, the 
network metric system 10 preferably also employs password protection in order to limit 
access as desired (e.g., access to management applications). 

Type of Service 

A preferred embodiment to the present invention also provides users with the ability 
to define multiple service types prepare of nodal members 30. For example, a user is able to 
specify TCP/UDP Port, DiffServ (differentiated services) field bit values, payload (zeros, 
ones, and random), and packet length for each user-defined service type. This type of quality 
of service specific behavioral information is then readily available in the system reports. 
Further, the work stations and preferred embodiments of the present invention also allow 
vectors to be disabled without being deleted from the database 40. This provides the 
advantage of saving a user from having to redefine a previously defined vector. 

Certain networks support different priority levels for the routing of network traffic. 
These policies can be based on the type of service (TOS) field settings in a packet or they can 
also be based on other parameters such as the source address, packet contents, port number, 
or other header information. TOS field or differential services settings indicate data delivery 
priority. This priority may or may not be ignored by the routers in the path to the receiving 
nodal member 30. Some routers may actually replace these settings with different ones. 
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For example, a router supports two policies, 'high priority' and 'best effort', with the 
default being best effort. The router knows by a packet's TOS field settings if the packet is a 
default best effort packet or a high priority packet. The router then schedules the packets 
transmitted based on the policy. For example, the router reserves 25% of the sending 
bandwidth for high priority packets and the rest of the transmitting bandwidth for best effort 
packets. Because TOS fields and other parameters that affect QOS can be modified it is 
possible to measure the different QOS policies and their effects. 

Measurement Sequence 

In a typical system packets are sent one at a time in a round robin fashion. In order to 
measure jitter, a minimum of 2 packets from a single vector must be transmitted in order with 
no other packets in between. The number of packets that are transmitted one after another in 
a vector is called the measurement sequence. This is also known as a burst. A measurement 
sequence size of one is equivalent to the normal round robin transmission scheme. This can 
be used if the jitter calculations are not desired. 

This embodiment of the present invention utilizes a round-robin measurement 
sequence. When multiple vectors are defined per nodal member 30, the measurement packets 
are transmitted in complete blocks, rather than interspersed with packets for other vectors. 
This guarantees accurate jitter measurements in the presence of multiple vectors. 

Bandwidth Allocation 

Another advantageous feature of the network metric system 10 of the present 
invention is its ability to provide user-definable measurement bandwidth allocation. This 
allows service providers that do not have a large amount of bandwidth available for 
measurement traffic to still be able to utilize the network metric system 1 0 of the present 
invention. In a preferred embodiment, the vector rates are automatically adjusted in order to 
utilize only a predetermined amount of bandwidth. Once the user decides upon the amount of 
bandwidth to be allocated for measurement traffic, each nodal member 30 in the network 
metric system automatically calculates the rate at which measurement packets are generated 
based on the number of vectors, packet size, and the bandwidth allocated. 

Test bandwidth is the rate at which packets for a vector are transmitted. Transmitted 
packets are not sent out all at once at the beginning of the measurement period. Instead 
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packets are transmitted out, based on measurement sequence, evenly spaced throughout the 
measurement period. The maximum test bandwidth depends on certain factors: Maximum 
bandwidth of the network; the number of vectors at work on the nodal member 30; the 
number of packets per measurement period per vector; the packet size per vector; the 
5 measurement period. 

The number of packets transmitted in a measurement period is definable per vector. 
The minimum number of packets is one. The maximum number of packets transmitted per 
vector is dependant upon: the test bandwidth; the number of vectors at work on the nodal 
member 30; the number of packets per measurement period per vector; the packet size per 
1 0 vector; the measurement period. 
2 Packet Size and Pavload 

Packet size is dependent upon the size of the Ethernet header, Ethernet CRC value, IP 
header, optional IP header, CQOS header and payload. The Ethernet header, Ethernet CRC 
value, IP header, and CQOS header are always the same size and this is the minimum size of 
15 a measurement packet. The maximum packet size is currently defined as the maximum size 
of an Ethernet packet. This size is currently equal to 1500 bytes total including the header. 
This size was chosen in order to try and eliminate further packet fragmentation by routers. 
~ This may be changed in the future. 

The size of the payload can be changed and is what determines the size of the packet. 
20 The minimum size of the payload is 0. The maximum size of the payload is: 

Maximum packet size - Ethernet header size - Ethernet CRC value size - IP header 
size - Optional IP header size - TCP/UCD header size (if used) - CQOS header size 

The contents of the payload can be specified as being filled with random numbers, all 
0's, or all 1 's. The random numbers for each packet are truly randomized and are not 
25 generated once for all packets transmitted. 
HDEFAULTS 

HDEFAULTS are the default values given for vector characteristics. Packet 
information HDEFAULTS are automatically chosen to populate the packet when configuring 
a vector. Values of this type include the contents of the CQOS and IP headers. These values 
30 also specify the payload contents of the packet. 
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Control information HDEFAULTS initially set the defaults for information regarding 
measurement sequence, test bandwidth, and any other information external to the 
measurement packets themselves. Preferably, users can modify these characteristics, if 
needed, to other valid values. HDEFAULTS and specific vector characteristics can be 
retrieved from a nodal member 30. This makes it possible to fill in the HDEFAULT values 
through an application before setting up a vector on nodal member 30. In a preferred 
embodiment of the present invention, the HDEFAULTS cannot be changed to other values. 

This section contains the HDEFAULT values defined and corresponding definitions 
in one preferred embodiment of the present invention. It will be appreciated that other 
defaults may be used in other embodiments of the present invention. Note that an unsigned 
-1 signifies that all bits in the field are set. 

//Payload Contents 

#define CVECTORCONFIGURATIONPAYLOADHDEFAULT 
0x4000000000000000 

#defme CVECTORCONFIGURATIONPAYLOADALLZEROS 0x01 
#defme CVECTORCONFIGURATIONPAYLOADALLONES 0x02 
#define CVECTORCONFIGURATIONPAYLOADRANDOM 0x04 
//IP Protocol 

#define CVECTORCONFIGURATIONIPPROTOCOLHDEFAULT 
(uintl6)-l 

//IP Protocol Types 

#define CVECTORCONFIGURATIONHEADERTYPEHDEFAULT 
0x4000000000000000 



#define CVECTOR_ 


CONFIGURATION 


HEADER, 


TYPE, 


NONE 


0x01 


#defme CVECTOR_ 


CONFIGURATION, 


HEADER, 


JYPE, 


UDP 


0x02 


#defme CVECTOR_ 


CONFIGURATION 


HEADER, 


_TYPE_ 


_TCP 


0x04 


#defme CVECTOR, 


CONFIGURATION 


HEADER, 


TYPE, 


RTP 


0x08 



//TCP Defaults 



33 

#define CVECTORCONFIGURATIONTCPFLAGSHDEFAULT 
(uintl6)-l 

#defmeCVECTOR_CONFIGURATION_TCP_WINDOW_SIZE_HDEFAULT 
(uint32)-l 

5 #defme CVECTOR CONFIGURATION TCP URGENT POINTER HDEFAULT 

(uint32)-l 

#define CVECTOR_CONFIGURATION_TCP_MSS_HDEFAULT 
(uint32)-l 

#define CVECTORCONFIGURATIONDEFAULTGATEWAYDHCP 0 
10 //Packet Size 

#define CVECTOR CONFIGURATION PACKET SIZE HDEFAULT 0 
//Burst Size 

#defme CVECTOR_CONFIGURATION_BURST_SIZE_HDEFAULT (uint64)-l 
//TTL 

15 #defme C VECTOR C ONFIGURATI ON TTL HDEF AULT (ushort)-l 

S //TOS 

#defme CVECTOR CONFIGURATION TOS HDEFAULT (ushort)-l 
//Optional IP 

#defme CVECTOR CONFIGURATION RECORD ROUTE HDEFAULT 
20 (uint8)-l 

#define CVECTOR CONFIGURATION SOURCE ROUTE TYPE HDEFAULT 
(uint8)-l 

#define CVECTOR_CONFIGURATION_SOURCE_ROUTE_TYPE_NONE 
(uint8)0 

25 #define CVECTOR_CONFIGURATION_SOURCE_ROUTE_TYPE_LOOSE 

(uint8)l 
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#defineCVECTOR_CONFIGURATION_SOURCE_ROUTE_TYPE_STRICT 
(uint8)2 

#defme CVECTOR_CONFIGURATION_SOURCE_ROUTE_COUNT_HDEFAULT 
(uint8)-l 
5 //Inactivity 

#defineCVECTOR_CONFIGURATION_INACTIVITY_TIMEOUT_HDEFAULT 
(uint64)-l 

//Outage 

#define CVECTOR CONFIGURATION OUTAGE TRIGGER HDEFAULT 
ID. (uint64)-l 

#define 

CVECTOR_CONFIGURATION_OUTAGE_RX_PACKET_COOL_COUNT_HDEFAULT 
* (uint64)-l 

//ARP 

15 #defme CVECTOR CONFIGURATION ARP RETRY COUNT HDEFAULT 

(uintl6)-l 

#defineCVECTOR_CONFIGURATION_ARP_TIMEOUT_HDEFAULT 
(uint64)-l 

//Ping 

20 #define CVECTOR_CONFIGURATION_PING_RETRY_COUNT_HDEFAULT 

(uintl6)-l 

#define CVECTOR CONFIGURATION PING TIMEOUT HDEFAULT 
(uint64)-l 

#define CVECTOR CONFIGURATION PING PACKET SIZE HDEFAULT 
25 (\iintl6)-l 

//Trace Route 

#define CVECTOR CONFIGURATION TRACE ROUTE PROBES HDEFAULT 
(uintl6)-l 
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#define 

CVECTORCONFIGURATIONTRACEROUTEMAXTTLHDEFAULT 
(uintl6)-l 

#define 

5 CVECTOR_CONFIGURATION_TRACE_ROUTE_WAIT_TIME_HDEFAULT 
(uint64)-l 

//General Definitions 
#define 

CVECTOR_CONFIGURATION_GONOGO_RETRY_COUNT_HDEFAULT 
10 (uintl6)-l 

#defme CVECTOR CONFIGURATION GONOGO TIMEOUT HDEFAULT 
(uint64)-l 

//Test Bandwidth 

#define CNODE CONFIGURATION TEST BANDWIDTH HDEFAULT 
15 (uint64)-l 

//Configuration 

- #defme CNODE_CONFIGURATION_IP_ADDRESS_DHCP 0 

#defme CNODE_CONFIGURATION_IP_ADDRESS_BOOTP 1 
#define CNODE_CONFI GURATION IP ADDRES S RARP 2 

20 #define CNODE CONFIGURATION SUBNET MASK DHCP 0 

#defme CNODECONFIGURATIONDEFAULTGATEWAYDHCP 0 
#define CNODE_CONFIGURATION_WAIT_TIME_HDEFAULT 
(uint64)-l 

#defineCNODE_CONFIGURATION_MEASUREMENT_PERIOD_HDEFAULT 
25 (uint64)-l 

//Connectivity 
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#defme 

CNODE_CONFIGURATION_IP_CONNECTIVITY_FREQUENCY_DELAY_HDEFAULT 
(uint64)-l 

#define 

5 CNODECONFIGURATIONIPCONNECTIVITYRETRYCOUNTHDEFAULT 
(uintl6)-l 

#define 

CNODE_CONFIGURATION_IP_CONNECTIVITY_TIMEOUT_HDEFAULT 
(uint64)-l 
10 Router Issues 

~ In modern routers there are two paths that can be taken when handling a packet, a 

slow path and a fast path. The slow path is taken if a packet requires special handling that 
cannot be handled directly by the hardware. In this case, the processor on the router must be 
involved to handle the packet. Conversely, the fast path is taken if a packet does not require 

1 5 special handling and does not have to be sent to the processor for handling. 

Events that can cause the packet to take the slow path include: TOS field settings that 
the router needs to modify; a packet size that is too large to be sent out without 

X fragmentation; and a packet with an optional IP header wanting record route or other routing 
information that must be extracted from the header. A side effect of this route path issue, is 

20 that a packet can be retransmitted with greater delay then packets that take the fast path. If 
this delay is long enough, this can cause packets to be received in the incorrect order, even if 
the packets are sent to the same router. 

The number of routers between the transmitter and receiver, called hops, can have an 
effect on certain results. As the number of hops increases, the chance of an increase in 

25 latency, jitter, and lost packets also increases. Latency and jitter may increase just because of 
the nature of receiving and retransmission. Lost packets may increase because the packet 
must go through a greater number of queues where most packets are dropped. 

Database 

Referring again to FIGURE 1 and the database 40 portion of the present invention, the 
30 network metric system 10 utilizes a database 40 that is SQL compliant. In a preferred 
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embodiment, the database 40 is an Oracle database that manages vector configuration 
information and all results. The raw data is stored and available for a variety of reports 70, as 
shown in FIGURE 4. Advantageously, since the reports 70are not pre-created, but rather are 
pulled directly from the database 40, based on user-defined parameters, the reports are 

5 flexible and reflect true averages for the time periods chosen. The averages can be considered 
true because they are not averages of averages, as commonly and mistakenly calculated by 
prior art measurement systems. A preferred database 40 of the present invention stores the 
original numerator and denominator data so that true averages can be calculated based on the 
user-defined parameters. The database 40 stores a full range of the complete set of Internet 

1 0 metrics that are described in detail below. Other data fields may also be added to the 

database 40 in other embodiments as desired. In one preferred embodiment, the network 
metric system 10 manages all aspects of the database 40. However, in other embodiments, 
the system 10 also supports unique data access requirements and customized application 
integration via the database. 

15 In a preferred embodiment of the present invention, the database 40 provides the 

vector configuration information to the service daemon 60, as well as storing measurement 
data transmitted from the nodal members 30 via the service daemon 60. The database 40 
obtains the vector configuration information from the user interface of the workstation 50 via 
- the application server 46. The application server 46 operatively connect the database 40 and 

20 the workstation 50 for system configuration and results display. Results display includes 
obtaining the results data from database 40 and preparing the data for display. 

Workstation 

Referring now to the workstation 50 portion of the network metric system 10 of the 
present invention, a browser based interface is utilized which allows CQOS management and 

25 reporting functions to be accessible from a simple web browser. As shown in FIGURE 1 , in 
one preferred embodiment the workstation 50 provides a user interfaces with the database 40 
through the application server 46 for system configuration. System configuration includes 
creating and sending vector configuration information to the database 40. In another 
embodiment of the present invention, the application server 46 is removed, and the 

30 workstation 50 interfaces with the service daemon 60. (In this embodiment, the functions of 
the application server 46 are performed by service daemon 60). 
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The network metric system 10 provides easy access to reports and management in the 
system from any computer without requiring special or complicated software installation. 
Preferably, the work station implements multiple secured access levels. Initial security levels 
include an administrator level and a user level. Preferably, the administrator has access to 
system configuration, which includes creation/modification/deletion of nodal members 30, 
vectors, service types, logical groupings of vectors, and the user access list. These functions 
are easily accessible to the administrator from the home page of the browser-based user 
interface. Typically, a user can only view reports. These multiple access levels allow a 
greater level of security to be implemented into the system. In a preferred embodiment of the 
network metric system 10, the user interface is secured using the Secure Socket Layer (SSL) 
protocol and the application server 46 also authenticates user connections. 

In a preferred embodiment of the network metric system 1 0, the workstation utilizes a 
traffic engineering application as an operations and analysis tool that provides a user interface 
to the network metric system 10. The primary function of the application is to provide 
meaningful presentation of network performance measurements in order to allow network 
planners to view real-time, large-scale, scientific measurement of the Quality of Service 
performance delivered by their IP networks. 

In one preferred embodiment to the present invention, the workstation 50 is utilized to 
implement user-definable groupings of vectors. Vectors can be logically grouped for ease of 
vector display and reporting. Useful groupings of vectors may include geographical, 
customer, network type, or priority based groupings. Additionally, groupings can also 
overlap (i.e., a vector can be part of several different groups). This configuration allows for 
ease of use and customizable reporting to suit various reporting needs and users. In some 
embodiments of the present invention, secure access may be available on a per-group basis. 

Alarms 

A preferred embodiment of the network metric system 10 provides customised alarms 
for automatic triggering and notification of emerging performance issues, including 
integration into Network Management Systems (NMS) to enhance a customer's own network 
operations facilities. User alerts may be viewed through the user interface 80 (as shown in 
FIGURE 5), and may activate notification functions such as e-mail, paging, or transmission 
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of SNMP traps for integration with established Network Management Systems (NMS) like 
HP OpenView. 

Furthermore, the alarm capability of the network metric system 10 offer a tangible 
method of dealing with Service Level Agreements (SLA) compliance. Through the use of 
several levels of alarm severity, set to trigger at thresholds progressively closer to the 
violation of a SLA, a Service Provider may proactively manage their service level agreements 
for exactly the conditions that cause non-compliance (e.g., delay or outages). 

The alarm capability and general measurement capability of the present invention 
allows grouping of measurement vectors to give additional SLA benefits. Groups create a 
method of applying hierarchies to measurement solutions. Through the use of groups, a 
customer may separate the measurement of their IP network many ways, while only 
instrumenting the measurement solution once. 

Reports 

Referring again to FIGURE 4, in one preferred embodiment of the network metric 
system 10 of the present invention, basic real-time reports 70 are automatically generated 
(without any additional configuration) that show one-way delay, jitter, packet loss and 
availability measurements. These results are preferably presented in a side-by-side graphical 
and tabular display, with a separate line for each service type. True averages are provided for 
each time period, and a minimum, maximum, and standard deviation are also automatically 
shown. The present invention produces results using numerator and denominator values, so 
that true averages can be calculated through a sum of all numerators and a sum of all 
denominators. This avoids the smoothing effect created by calculating an average of 
averages. 

A preferred embodiment in the present invention provides a wide array of reporting 
options. The system allows a user to designate continuous time or time period history 
reporting, measurement period, start time, end time, and bi- or uni-directional measurements. 
This type of flexible reporting with customizable time periods up to and including the current 
period is highly advantageous to a system user. The network metric system of the present 
invention preferably provides click through access to powerful results that are not available 
from prior measurement products or services. 
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General Algorithm Description 

In a preferred embodiment of the network metric system 10, after a vector has been 
configured on the transmitting nodal member 30, and the receiving nodal member has 
initialized a vector handler, the receiving nodal member is ready to receive measurement 
packets. Preferably, a linked list is created for each vector, for each measurement period. 
Measurement packets received from another nodal member 30 are stored in this linked list in 
the order received. Packets are stored in an atomic data unit structure. Hereinafter, CQOS 
measurement packets and atomic data units are considered equivalent. In one preferred 
embodiment of the present invention, after the measurement period is over, 2 minutes are 
given for any straggling packets to arrive. When the vector receives the ending measurement 
period packet, the result calculation routines are called. In one preferred embodiment of the 
present invention, if the end of measurement period packet is not received within 48 hours, 
the results are discarded. 

In one preferred embodiment of the network metric system 10, the calculation 
methods take the packets received and fills out the results. The results are then sent to 
another computer for subsequent analysis. The memory associated with the vector's current 
measurement period is then freed. The following sections describe elements of the algorithm 
in more detail. 

Identification 

In a preferred embodiment of the present invention, as each packet is received, the 
packet is inserted into the appropriate linked list based on identification information 
contained in the CQOS header. This identification information is made up of four fields, the 
sendingnodal memberlD, the sendingCVectorlD, the measurementPeriodID; and the nodal 
memberMeasurementPeriodlD. 

The sendingnodal memberlD is a unique identifier that is given to each nodal 
member 30. The sendingCVectorlD is the vector identifier that is unique per sending nodal 
member 30. The measurementPeriodID is an identifier starting from 0 assigned to each 
measurement period. The nodal memberMeasurementPeriodlD is also an identifier assigned 
to each measurement period, but if differs from the measurementPeriodID in that it is unique 
and not 0 based. Based on 3 of the 4 identifiers, that is, sendingnodal memberlD, 
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sendingCVectorlD, and nodal memberMeasurementPeriodID, a guaranteed unique linked list 
is located to place the incoming packets into. 
Sorting 

In a preferred embodiment of the network metric system 10, it is possible that packets 
received in an order which is different than how they were transmitted. This usually indicates 
that some packets took different routes than others. This can also happen if certain packets 
require special handling that causes some packets to take the slow path instead of the fast path 
on a router. In any case, the packets received must be sorted into their original transmitted 
order because of jitter calculation requirements. Three special cases need to be dealt with 
when sorting: duplication, dropped packets, and fragmentation. 

Duplicates 

Duplicate packets can occur because of various reasons. Duplicate packets are taken 
into account for most result calculations, except for jitter, outages, and ordering. In these 
cases, only the first occurrence is used. In order to detect duplicates, the list is traversed and 
all other items in the list are compared with the current item. If the sequence number of the 
item and the transmitted timestamp match, then there is a duplicate. The index of the item is 
placed in an array allocated to store duplicate indexes. The current item is then incremented 
to the next one until all items in the list have been checked. Note that all items are placed in 
the duplicate array, even the first occurrence thereof. 

Further along in the algorithm, the total number of duplicates, minimum number of 
duplicates for one item, and maximum number of duplicates for one item are all calculated 
based on the duplicate array. These are stored in the results as packetsDuplicated, 
packetsDuplicatedMin, and packetsDuplicatedMax. Eventually, an extra metric may be 
added that counts duplicates that took a different route from one another using TTL value 
comparisons. 

Dropped Packets 

A packet is dropped when a packet is transmitted, but it is not received. The 
definition of the number of dropped packets is: 

(# packets transmitted - # packets received) - # of duplicate packets 
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The number of packets transmitted is sent along with the special packet at the end of 
the measurement period. By counting the number of packets in the linked list, the number of 
packets received is known. When sorting the packets, a list of duplicate packets is built up so 
that the number of duplicate packets is known. With this information, the formula can be 
applied and the results saved in the packetsDropped field. 

Fragmentation 

Fragmentation occurs in routers when a packet arrives that cannot be sent out on the 
next route without breaking the packet up into smaller pieces. This typically occurs because 
the next part of the route uses a protocol that has a maximum packet size that is smaller than 
the size of the current packet. Currently, the maximum size of the packet is set to the 
maximum size of an Ethernet packet (1500 bytes). To calculate the fragmentation results, a 
loop is used to retrieve the proper results from all of the atomic packet data. 

In accordance with the present invention, packetsFragmented is the sum of all of the 
packets that were fragmented and packetsFragmentedMin, packetsFragmentedMax, 
packetsFragmentedAverageNumerator, packetsFragmentedAverageDenominator are the 
minimum, maximum, and average fragmented packets respectively. 

Hop Count OTP 

Hop count or Time To Live (TTL) is the maximum number of routers that can be 
traversed when transmitting data. Each time a packet is retransmitted by a router, it's TTL 
value is reduced by one. A router that receives a packet with a TTL value of 0 drops the 
packet. The transmitting nodal member 30 saves the original TTL value in the CQOS header 
so that when the packet arrives the hop count can be calculated. The HDEFAULT value of 
TTL is the maximum, 255. 

To calculate the TTL results, a loop is used to retrieve the proper results from all of 
the atomic packet data. The current packet's TTL value is temporarily stored so that if the 
TTL field is different for the next packet, the number of changes can be saved. This indicates 
that the packet took a different route than the previous packet. 

In the present invention, packetsTtlMin, packetsTtlMax, packetsAverageNumerator 
and packetsAverageDenominator are the minimum, maximum, and average TTL values. 
packetsTtlChanges is the number of changes of TTL values between all of the packets. 
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Jitter 

Jitter is the difference between the time a packet is expected to arrive, and the time it 
actually arrives. In other words, a measurement sequence of packets is transmitted one 
second apart. Jitter is how far apart the packets actually arrived. Jitter is mathematically 
5 defined as: 

|(Tx-Rx)| = the absolute value of the time transmitted minus the time received. 

To measure jitter, a measurement sequence greater than one must be transmitted and 
received. In addition, the received list of packets must be sorted into transmitted order before 
calculating jitter. To calculate jitter the packets are traversed in transmitted (sorted) order. 
10 For each measurement sequence, the first packet in the measurement sequence is used as a 

base. The remaining packets in the measurement sequence use the previous packet's received 
~ and transmitted timestamps and subtract them from their own to calculate the jitter. 
~ Dropped packets are not counted in jitter calculations. For example, if a burst of 5 

packets comes in and packet 3 is dropped, the transmitted sequence of packets that were 
1 5 actually received is: 1 ,2,4,5. The jitter between packets 1 ,2 and the jitter between packets 4,5 
will be calculated. But since packet 3 was dropped, the jitter between packets 2,3 and 3,4 
will not be calculated and included in the results. 
^ The accumulated jitter, minimum jitter, maximum jitter, sum of squares, sum of 

cubes, jitter count, and jitter burst count are all calculated and saved in jitterStdDevSums, 
20 jitterMin, jitterMax, jitterSumSqrd, jitterSumCubed, jitterCount, burstsReceived, 
respectively. 

Latency 

Latency is the amount of time that a packet takes to travel from the transmitter to the 
receiver: 

25 Rx — Tx = Timestamp received — Timestamp transmitted 

The timestamp when the packet is transmitted is placed in the packet in the CQOS 
header upon transmission. When the packet is received another timestamp is recorded. 

All the packets are traversed and the average latency, maximum latency, minimum 
latency, sum, sum of squares, sum of cubes, and number of latencies used for calculation for 
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all packets with non-corrupted CQOS headers are calculated. These values are saved in the 
latency AverageNumerator, latency AverageDenominator, latencyMin, latencyMax, 
latencyStdDevSums, latencyStdDevSumOfSquares, latencyStdDevSumOfCubes, and 
latencyStdDevN fields, respectively. 

5 Outages 

An outage occurs when a vector is not available. The causes of an outage can vary 
from a cable not correctly plugged in, to a router or network failure. In terms of 
measurement, an outage is determined if there are no measurement packets received within a 
certain time period. This period is set by default to be 10 seconds. However, any defined 

10 time period may be used in other embodiments of the present invention. If even 1 

measurement packet arrives within this set time period, then no outage will occur. Also, only 

_u the first occurrence of a duplicate counts towards a received packet. The remaining 

duplicates do not reset the outage counter. Therefore, packets with errors in them do not reset 
7 the counter. The timestamp of when a packet is received is currently used to calculate 

fS outages. 

= _ The outage algorithm works by looping through all of the packets received. Starting 

from the beginning of the received packets, the outage algorithm finds a packet without errors 
J and with no duplicates for it in the list, and saves the received timestamp of the packet. For 
f every packet, except for the first, the outage algorithm subtracts the time of the current valid 
20 packet received from the last packet's received timestamp. If the difference is greater than 
the outage trigger time (currently 10 seconds) then an outage has occurred and is recorded. 
The algorithm also looks for the last packet received to see if there is an outage of which it 
can compute the length, without using the maximum of the remainder of the measurement 
period. 

25 The result of the algorithm is the sum of all outage durations, the minimum outage 

duration, the maximum outage duration, and the number of outages. These values are saved 
in the results as: outageDurationTotal, outageDurationMin, outageDurationMax, and 
outages, respectively. 
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Ordering 

The order in which packets are received (as opposed to how they are is transmitted) is 
another set of data saved in a preferred embodiment of the present invention. To determine 
ordering metrics, an algorithm is applied whose purpose is to determine how many items are 
5 out of order. The algorithm distinguishes between individual packets and groups of packets. 
A group of packets is one in which all items in the group are in sequential order with no out 
of order packets there between. The end result of the algorithm is the number of groups of 
packets and the number of individual packets out of order. These results are stored in the 
rxGroupsOutOfOrder and rxPacketsOutOfOrder fields. 

10 In a preferred embodiment of the network metric system 1 0, enough RAM is used to 

hold a flag to represent each item in the list for which "presortedness" is to be determine. In 
one embodiment, this a bit or a byte array, with each having a size or speed advantage, 
respectively. The algorithm performs the following tasks: 

1 . Mark any strings at the beginning or end of an array that are already in 
15 position. 

A) Search array items, that have not been marked as moved, for the minimum and 
maximum number of items in the array. 

*t B) Examine the first unmarked item in the list (maintain an index to this item) to see if 

it is the minimum. 

20 If it is the minimum, then compute the length of the string which is already in place in 

order to simply mark the item as moved without counting the item. 

Examine each consecutive item. If this item is item [-1]+1 then move on to the next 
one. However, if this item is greater than [-1]+1, search the entire array of unmarked items 
for one which is in between these two items. If found, the end of the string is found, and all 
25 these items must be marked as moved without counting them. A value lower than the 

previous value will also terminate the string. If no value is found in-between, then the sting 
continues. 

C) Perform Step B again, except now moving from the end of the array. 

2. Next, in order to move the smallest runs first, start a variable called automark 
30 set to 1 . This means that as the array is searched for run lengths. If a run is found of length 1 , 
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the run is marked immediately as moved, and then counted. This variable is set to the next 
smallest run length found after searching the array for all runs of automark size. This 
prevents searching for unused run lengths on the next scan. 

3. After every run is moved, the algorithm transforms the new first or last 
5 unmarked item in the array from being out of position to being in position. This will only 
happen if either the run has a min or max value equal to the min or max value of the array, or 
if the string being moved has been moved from either the beginning or end of the array. If 
either is the case, then perform either 1(B) or 1(C) above, respectively. 

Example 1: 

10 10 11 12 13 45 4647 14 15 7 1 23 45 6 

Found 7, mark as moved and count. Found 14 15, mark as moved and count. Since 
14 and 15 are marked as moved, 10-47 will now be viewed as one long string. Thus, 
process 1-6 next. Since this string contains the min value, check the first unmarked item in 
I the array now for min (10). Since it is the min, mark as moved without counting. Everything 
IS is in order 3 groups of 9 items. 

For the following example MMC = Mark as Moved and Count 

MMDC = Mark as Moved and Don't Count 

Example 2: 
13246587 

20 1 MMDC. 3 MMC. 2 4 MMDC. 6 MMC. 5 MMDC. 8 MMC. 7 MMDC. 3 groups 3 

items. 

Example 3: 

15 40 48 1 12 16 17 18 3 4 5 6 7 8 47 

15 MMC. 40 MMC. 48 MMC. 47 MMDC. 1 MMDC. 12-18 MMC. 3-8 MMDC. 4 
25 groups 7 items. 

Example 4: 

41 4243 154048 1 12 16 17 1834567847 
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Find 15 MMC. Find 40 MMC. Find 48 MMC. Since 48 max check end string 47 in 
position MMDC. Find 1 MMC. 41-43 found MMC. Find 12-18 MMC. 

Port Counters 

Port counters are used to keep track of the number of frames, collisions, and certain 
5 types of errors calculated by the 'layer 2' (Ethernet layer) interface. Each data packet in the 
received list contains a running estimate of these items. The estimates in the first packet are 
subtracted from the estimates in the last received packet and these are stored as results for the 
measurement period. 

The items saved are: 
1 0 Number of good frames transmitted - estimate_txGoodFrames 

Number of transmitted packets with collisions - estimatetxCollisions 

Number of transmitted packets with no collisions - estimate_txNoTxCollisitons 

Number of good frames received - estimate_rxGoodFrames 

There are also various error values that are stored. These are discussed later in the 
1 5 Error Handling / Ethernet Errors sections. 

Optional IP Fields 

Internet Protocol supports an optional header field that has numerous uses, of which 
the nodal member 30 currently supports three. Internet Protocol can be used to record the 
route a packet traversed, or specify loosely or very strictly the way a packet should be routed. 
20 The maximum size of this field is specified as 60 bytes. Due to the maximum size limitation, 
the maximum number of IP addresses that can be recorded or specified is 9. 

When recording the route, up to 9 routers (the first 9 routers) the packet traversed are 
saved in the header. Any other routers visited are not be recorded. In order to record this 
information, it is necessary for the packet to take the slow path through the router. With a 
25 strict route specified, the IP addresses of the routers in the optional field must be traversed 
exactly as placed in the optional header. Using a loose route, the IP addresses of the routers 
in the optional field must be traversed, but they can be visited in any order and with 
additional router visits there between. 
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The first and last packets that are received are checked for optional IP header 
information. Whether or not they contain this information, the packet identifiers of the first 
and last packet are saved in the firstRoutePacketID and lastRoutePacketID fields. The type of 
optional IP header information is saved in the firstRouteType and lastRouteType fields. The 
values contained therein are defined as: 

CQOS_RESULTS_ROUTE_TYPE_NONE, 
CQOS_RESULTS_ROUTE_TYPE_RECORD, 

CQOS_RESULTS_ROUTE_TYPE_LOOSE, 

CQOS_RESULTS_ROUTE_TYPE_STRICT 

The actual optional header information and route count for the first and last packet is 
stored in the firstRouteData, lastRouteData and firstRouteCount, lastRouteCount. 

Ethernet Errors 

The first set of errors involves errors that were found previously at the Ethernet layer. 
These 'layer 2' errors are summed in each of the appropriate fields in the results for all 
packets received in the measurement period. These errors are: 

The number of CRC errors caused by a bad checksum - rxCRCErrors 

Alignment errors - rxAlignmentErrors 

Frame too short errors - rxShortFrameErrors 

Frame too long errors - rxLongFrameErrors 

Total received errors — rxErrors 

In addition, the estimates of certain errors in the first packet received are subtracted 
from the estimates in the last packet received. These are stored as results for the 
measurement period. These 'estimate' values are: 

The number of CRC errors caused by a bad checksum - estimate_rxCRCErrors 

Alignment errors - estimate_rxAlignmentErrors 

Frame too short errors — estimate rxShortFrameErrors 

Resource errors — estimate_rxResourceErrors 
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COOS Header Errors 

The next set of errors involves the CQOS header checksum. This checksum is a 64 
bit value that validates the CQOS header items. If this checksum is incorrect, critical data 
cannot be retrieved from the packet such that it cannot be used for TTL, IP protocol, TOS, 
latency, outage, and jitter calculations. The IP payload is also considered corrupted since the 
CQOS header is part of the IP payload. If the CQOS header is corrupted, the packet is not 
stored in the array of packets used for further computations and is ignored for the metrics 
mentioned below. These items are stored in the array of packets used for further calculations: 

The received timestamp - rxTimestamp; 

The transmitted timestamp - txTimestamp; 

The identifier of the current packet in order transmitted - sequence; 
The identifier of the burst - cBurstID; 
A pointer to the packet — packet; and 

A general error value that signifies if there is a layer 2, IP, payload, or CQOS header 
error- errored. 

These items cannot be calculated for the packet if the CQOS header is corrupted: 

The identifier of the period - CperiodID (only one packet received in the 
measurement period has to be free of CQOS header errors to get this anyway); 

The number of TTL changes — packetsTtlChanges and all other TTL results; 

The number of protocol changes - packetsIPProtocolChanges; 

The number of TOS field changes - packetsTosChanges and all other TOS results; and 
The latency, jitter, and outages - (depends on rxTimestamp and txTimestamp). 
The following results are incremented with each corrupted CQOS header found: 
The number of corrupted CQOS headers - packetsCQOSInfoCorrupted; and 
The number of payloads corrupted — packetsPayloadCorrapted. 
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IP Header Errors 

The last set of errors involves the IP header checksum. This checksum is a 16 bit 
value that validates the IP header items. The checksum does not validate the payload that 
includes the CQOS header. The following items cannot be properly computed or stored if the 
5 IP header is corrupted and so the packet is skipped for these calculations: 

The number of fragmented packets - packetsFragmented and all other fragmentation 

results; 

The number of TTL changes - packetsTtlChanges and all other TTL results; 

The number of protocol changes - packetsIPProtocolChanges; and 
10 The number of TOS field changes - packetsTosChanges and all other TOS results. 

: The following results are incremented with each corrupted IP header found: 

The number of corrupted IP headers - packetsIPCorrupted; and 

The number of corrupted optional IP headers - packetsOptionalHeaderCorrupted. 
* Other Info 

15 Additionally, there are a few other miscellaneous items that are stored in the results. 

bytesReceived is the sum of the number of bytes received in total for the measurement 
period. To calculate the bytesReceived, the packets are traversed and all of the bytes received 
for each packet are summed. 

Up to the first 10 TOS fields are saved into the packetsFirstlOTos array. To find the 
20 TOS fields to store, all received packets with valid CQOS and IP headers are traversed. The 
values in the TOS fields in the CQOS header and IP header are examined. The values are 
compared and, if they differ, the TOS field setting is saved in the packetsFirstlOTos array. 
This indicates a router modified the TOS field before re-transmitting the packet. The number 
of changes stored is placed in packetsFirstlOTosCount. 

25 Some general vector information is also stored in the results. The packetsTransmitted, 

bytesTransmitted, measurementPeriodNanoseconds, and universalTime are retrieved from the 
vector itself and saved. 

Version information is stored in the results. This information consists of: 
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Transmitting and receiving main versions - txMainVersion, rxMainVersion; 

Transmitting and receiving Big Joe versions - txBigjoeVersion, rxBigjoe Version; 

Transmitting and receiving FPGA versions - txFPGAVersion, rxFPGA Version; and 

Transmitting and receiving Mercury versions — txboardVersion, rxboardVersion. 

5 Transmitting and receiving nodal member 30 temperature information is saved in the 

results. The minimum, maximum, average temperatures of the transmitting and receiving 
nodal member 30 are saved in: 

txtemperatureMin, rxtemperatureMin, txtemperatureMax, rxtemperatureMax, 
txtemperatureAverageNumerator, rxtemperatureAverageNumerator, 
10 txtermperatureAverageDenominator, and rxtemperatureAverageDenominator 

Results Structure 

The results structure and the elements that comprise the results structure are 
referenced below, and are used to store all results calculated by the measurement algorithms. 
In one preferred embodiment of the present invention, reference to the result structure is a 
15 reference the structure below. 

struct struct_cqosResults { 

//note all counters are in terms of measurement period 
//warn R/S route info 
//reserved info 
20 uint32 version; 

uint64 sendingnodal memberlD; //ID unique for each nodal 

member 

uint64 sendingCVectorlD; //ID of vector unique for each nodal 

member 

25 uint64 measurementPeriodID; //ID for measurement period (0 based) 

uint64 nodal memberMeasurementPeriodID; //ID for measurement period (unique, 
not 0 based) 
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uint64 bitMask; //CQOS_RESULTS_CONFIG0_* 

uint64 universalTime; //current UTC time 

uint64 measurementPeriodNanoseconds; //length of measurement period in nsec 



uint64 burstsReceived; 

5 uint64 bytesReceived; 

uint64 bytesTransmitted; 

uint64 packetsReceived; 

uint64 packetsTransmitted; 

uint64 packetsOutOfOrder; 

10 uint64 packetsDuplicatedMin; 

uint64 packetsDuplicatedMax; 

uint64 packetsDuplicated; 
- (numerator) 

uint64 packetsDuplicatedDenominator; //# of packets that had 1 or more 
J5 duplicates (denominator) 

~ uint64 packetsDropped; 

uint64 packetsFragmented; 

uint64 packetsFragmentedMin; 

uint64 packetsFragmentedMax; 



//sum of bursts received 
//sum of bytes received 
//sum of bytes transmitted 
//sum of packets received 
//sum of packets transmitted 
//# of packets out of order 
//min # of duplicated packets 
//max # of duplicated packets 
//# of packets with duplicates 



//# of packets dropped 
//# of packets that were fragmented 
//min # of fragmented packets 
//max # of fragmented packets 



20 uint64 packetsFragmentedAverageNumerator; //avg. # of fragmented packets 

(numerator) 

uint64 packetsFragmentedAverageDenominator; //avg. # of fragmented packets 
(denominator) 

uint64 packetsIPCorrupted; //# of packets with corrupted IP header 

25 uint64 packetsCQOSInfoCorrupted; //# of packets with corrupted CQOS 

header 
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uint64 packetsPayloadCorrupted; //# of packets with corrupted payload 
(includes CQOS header) 

uint64 packetsOptionalHeaderCorrupted; //# of packets with corrupted optional IP 
header/* WARN SYNC */ 

uint64 packetsTtlChanges;//# of packets where TTL changed from previous packet 

uint64 packetsTtlMin; //min#ofTTL 

uint64 packetsTtlMax; //max # of TTL 

uint64 packetsTtlAverageNumerator; //avg. # of TTL value (numerator) 

uint64 packetsTtlAverageDenominator; //avg. # of TTL value (denominator) 

uint64 packetsIPProtocolErrors; //# of IP protocol errors 

uint64 packetsIPProtocolChanges; //# of packets where IP protocol changed 
from previous packet 

uint64 packetsTosChanges; //# of packets where TOS field changed 
from previous packet 

int packetsFirst 1 0TosCount;//# of items saved in the packetsFirst 1 OTosCount 

field 

int packetsFirstl 0Tos[l 0]; //1st 10 TOS fields that were changed during 
transmission 

/* Jitter */ 

uint64 jitterMin; //min jitter # 

uint64 jitterMax; //max jitter # 

uint64 jitter AverageNumerator; //avg. jitter # (numerator) 

uint64 jitter AverageDenominator; //avg. jitter # (denominator) 

uint64 jitterStdDevSumOfSquares; //jitter sum of squares 

uint64 jitterStdDevSumOfCubes; //jitter sum of cubes 

uint64 jitterStdDevSums; //sum of jitter 

uint64 jitterStdDevN; //# of jitter calculations 
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/* Latency */ 




uint64 latency TimestampsMismatch; 


//# of cases where rx < tx 


uint64 latencyMin; 


//min latency # 


uint64 latencyMax; 


//max latency # 


uint64 latency AverageNumerator; 


//avg. latency # (numerator) 


uint64 latency AverageDenominator; 


//avg. latency # (denominator) 


uint64 latencyStdDevSumOfSquares; 


//latency sum of squares 


uint64 latencyStdDevSumOfCubes; 


//latency sum of cubes 


uint64 latencyStdDevSums; 


//sum of latency 


uint64 latencyStdDevN; 


//# of latency calculations 


/* Outages */ 




uint64 outages; 


//# of outages 


uint64 outageDurationMin; 


//min length of outages in nsec 


uint64 outageDurationMax; 


//max length of outages in nsec 


uint64 outageDurationTotal; 


//total outage length in nsec 


int firstRouteType; 


//IP optional type for first packet 


int firstRouteCount; 


//# of IP addresses in firstRouteData 


uint64 firstRoutePacketID; 


//sequence # of first packet 


uint3 2 fir stRouteData[9] ; 


//IP optional IP addresses 


int lastRouteType; 


//IP optional type for last packet 


int lastRouteCount; 


//# of IP addresses in lastRouteData 


uint64 lastRoutePacketID; 


//sequence # of last packet 


uint3 2 lastRouteData[9] ; 


//IP optional IP addresses 


/* Layer 2 Counters */ 




uint64 rxCRCErrors; 


//# of checksum errors 
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uint64 rxAlignmentErrors; 
uint64 rxFrameTooShortErrors; 
uint64 rxFrameTooLongErrors; 
uint64 rxErrors; 
/* Other Info*/ 
uint32 txSystemType; 
uchar txMainVersion[8]; 
uchar txBigjoeVersion[8]; 
uchar txFPGAVersion[8]; 
uchar txBoardVersion[8]; 
uint32 rxSystemType; 
uchar rxMainVersion[8]; 
uchar rxBigjoeVersion[8]; 
uchar rxFPGAVersion[8]; 
uchar rxBoardVersion[8]; 
uint64 txTemperatureMin; 
uint64 txTemperatureMax; 



//# of alignment errors 
//# of frame too short errors 
//# of frame too long errors 
//The above errors plus all others 

//system type of transmitter 
//code version of transmitter 
//Big Joe version of transmitter 
//Big Joe FPGA version of transmitter 
//Big Joe board version of transmitter 
//system type of reciever 
//code version of reciever 
//Big Joe version of reciever 
//Big Joe FPGA version of receiver 
//Big Joe board version of receiver 
//min temp of transmitter 
//max temp of transmitter 



uint64 txTemperatureAverageNumerator; //avg. temp of transmitter (numerator) 

uint64 txTemperatureAverageDenominator;//avg. temp of transmitter 
20 (denominator) 

uint64 rxTemperatureMin; //min temp of receiver 

uint64 rxTemperatureMax; //max temp of receiver 

uint64 rxTemperatureAverageNumerator; //avg. temp of reciever (numerator) 

uint64 rxTemperatureAverageDenominator; //avg. temp of reciever (denominator) 

25 /* Port Counters */ 
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uint64 estimate_txGoodFrames; 

uint64 estimate_txCollisions; 

uint64 estimate_txNoTxCollisionErrors; //transmitted late collision errors + max 
collision errors (82559:) 

uint64 estimate_rxGoodFrames; 

uint64 estimate_rxCRCErrors; 

uint64 estimate_rxAlignmentErrors; 

uint64 estimate_rxResourceErrors; 

uint64 estimate_rxShortFrameErrors; 

/* Out of Order Counters */ 



//transmitted good frames 
//transmitted collisions 



//received good frames 
//received checksum errors 
//received alignment errors 
//received resource errors 
//received short frame errors 



uint64 rxPacketsOutOfOrder; 
uint64 rxGroupsOutOfOrder; 



//# of packets out of order 
//# of groups out of order 
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Atomic Packet Data Structure 

In one preferred embodiment of the present invention, this structure is used to store 
information for each measurement packet received. A linked list of these structures for the 
current measurement period is located initially by the measurement algorithms. The list is in 
order received. 

struct struct_cqosAtomicPacketData { 

uint8 cqosheader lPProtocol; 

uint8 cqosheader TOS; 

uint8 cqosheader TTL; 

uint64 cqosheader_nodal memberlD; 



//original IP protocol 
//original TOS 
//original TTL 

//ID unique for each nodal 



uint64 cqosheader_nodal memberPeriodID;//ID for measurement period (unique, 
not 0 based) 



uint64 cqosheader_CVectorID; //ID of vector unique for each nodal 

member 





uint64 cqosheaderCPeriodID; 


//ID for measurement period (0 ba 




uint64 cqosheader BurstID; 


//ID for the burst packet is in 


5 


uint64 cqosheader PacketID; 


//IP of packet (sequence) 




uint64 cqosheader_TxTimestamp; 


//Timestamp when transmitted 




/* Non-CQOS Header derived information */ 




uint32 sourcelP Address; 


//IP address of transmitter 




ushort bytesReceived; 


//Total bytes received -> Includes i 


id 


ushort numberOfFragments; 


//Fragments of original packet? 




uint8 optionalHeaderType; 


//UDP/TCP (IP Protocol #) 




uint8 ipProtocol; 


//recieved IP protocol 




uint8 ttl; 


//received TTL 




uint8 tos; 


//received TOS 


15 


uint64 rxTimestamp; 


//received timestamp 




uint8 xxRoute; //IP optional route info,0 no rr/sr/lr. Else 0x7=rr, 0x83=lr, 



0x89=sr 

uint8 recordRouteCount; //number of routes recorded if option 

used 

20 uint32 recordRouteInformation[9]; //route info if option used 



/* Port Counters */ 

uint64 estimate_txGoodFrames; //# of transmitted good frames 

uint64 estimate_txCollisions; //# of transmitted collisions 

uint64 estimate_txNoTxCollisionErrors; //# of transmitted late collision errors + 
25 max collision errors 

uint64 estimate_rxGoodFrames; //# of received good frames 
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uint64 estimaterxCRCErrors; 


//# of received CRC errors 


uint64 estimate_rxAlignErrors; 


//# of received alignment errors 


uint64 estimate_rxResourceErrors; 


//# of received resource errors 


uint64 estimate_rxShortFrameErrors; 


//# of received short frame errors 


/* Packet Error Counters */ 




uint32 12_rxCRCError:l; 


//packet has level 2 checksum error 


uint32 12_rxAlignmentError: 1 ; 


//packet has level 2 alignment error 


uint32 12_rxFrameTooShortError: 1 ; 


//packet has level 2 frame too short error 


uint32 12_rxFrameTooLongError:l; 


//packet has level 2 frame too long error 


uint32 12_rxError:l; 


//packet has level 2 error 


uint32 cqosheader_PayloadChecksum6k: 1 ; //packet payload checksum ok 


uint32 cqosheader_HeaderChecksumOk:l; //packet CQOS header checksum ok 


uint32 ipHeaderChecksumOk:l; 


//packet IP header checksum ok 


uint32 ipHeaderMiscError: 1 ; 


//packet IP misc error 


uint32 ipProtocolErronl; 


//packet IP protocol error 


uint32 optionalHeaderChecksumOk:l; 


//packet IP optional header error 


uint32 errored:!; 


//packet general error 



/* House Keeping Information */ 

pATOMICPacketData next; //pointer to next packet 



Calculation Packet Data Structure 

An array of these structures is computed from the original list of AtomicPacketData 
structures by the measurement algorithms. This list is used to eliminate packets with any 
CQOS header errors and make it easier to reference the packets without traversing the list 
each time. 

struct struct recordlnfo { 
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uint64 rxTimestamp; //timestamp recieved 

uint64 txTimestamp; //timestamp transmitted 

uint64 sequence; //sequence number (ID) of the packet 

uint64 cBurstID; //burst ID number 

5 int errored; //error flag 

int duplicatedCounted; //duplicate already counted flag 



pATOMICPacketData packet; //pointer to ATOMICPacketData item 

>; 

System Operation 

19/ The logical operations of a preferred embodiment network metric system 10 of the 

present invention utilize the components of the system in a logical sequence. In a preferred 
c embodiment of the network metric system 10 of the present invention, a vector is the 
fundamental measurement unit. A vector is defined as a packet type and source and 
destination pair. The packet type describes what the characteristics of the packet are. All 

15 packets for the vector have the same characteristics (i.e. packet types.) Packet types include 
^ the ability to control: length of packet; payload type (all zero's, all one's or random); header 
type (udp, TCP, none); udp/tcp source and destination port numbers; TTL value; 
TOS/DiffServ bits; IP protocol value; IP loose, strict, and record route options; Default 
gateway to use for routed networks; Source and Destination addresses; and TCP Header 

20 information such as [window size, MSS option, FLAGS, urgent pointer]. 

The format of the packet is as follows: 















Src 


Dest 


c 


CQOS 


Payload 


Optional 


ip 

options 


IP 


Addr 


Addr 


R 
C 


Header 




UDP/TCP 


Header 












Header 











25 A vector is created by the service daemon 60. The service daemon 60 reads the 

configuration parameters of the vector from a database and communicates with the nodal 
member 30 via CQOS Protocol to create the vector on the sending nodal member 30. If the 
nodal member 30 accepts the configuration request, the nodal member responds to the service 
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daemon 60 with an "ok" status. If the nodal member 30 does not accept the configuration 
request, the nodal member will not create the vector and responds with an error status. Once 
the vector is created on the sending nodal member 30, the service daemon 60 issues the 
Readiness test command (via CQOS Protocol). The readiness test includes a set of tests 
5 including the Go/NoGo test, as previously discussed. 

Once again, the tests included are: 

(1) ARP Default Gateway: Send an ARP (Address Resolution Protocol) request to the 
gateway and store into memory round-trip time (RTT) of the ARP request, execute time, the 
IP address of the default router, and the MAC address (if valid response) of the default 
10 address; 

" (2) Ping Default Gateway: Ping (I CMP Echo Message) to the gateway and store into 

= memory the RTT time, execute time, and IP Address of the gateway; 

~ (3) Ping Receiving nodal member: Ping the receiving (destination) nodal member 30 

and record the RTT time, execute time and IP address of the receiving nodal member 30; 

15 (4) Trace Route to Receiving nodal member: Trace Route to the receiving nodal 

:.: member 30 and record each hops IP address, RTT. Also record the execute time and 
- destination IP address; and 

(5) Go/NoGo to Receiving nodal member: A message with the parameters of the 
vector, user ID, and password are sent to the receiving nodal member 30 asking for 

20 permission to make measurements. The receiving nodal member 30 looks at the parameters 
and compares the user ID and password with an Access Control List (ACL) maintained 
within the receiving nodal member 30. If the parameters are ok, and the user ID and 
password matches with a valid ACL entry, then the receiving nodal member 30 responds with 
a GO confirmation. Once the GO confirmation is received by the sending nodal member 30, 

25 then measurements start on the next measurement period (5 minute boundary). If the 

receiving nodal member 30 does not accept the parameters or user ID/password combination, 
then either NO response is be given to the sending nodal member 30, or a NoGo message is 
sent. In either negative case, the sending nodal member 30 will not under any circumstances 
send measurement packets. The feature is for security in that the users can not create vectors 
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to systems other than nodal members 30 nor create vectors to nodal members 30 that they do 
not control. 

Once the GO confirmation is received by the sending nodal member 30, then 
measurement packets are sent, which are formed as shown above. The number of packets 
5 sent is based on the number of total vectors within the sending nodal member 30, the 

characteristics of those vectors (e.g. packet size, packets / sequence) and the measurement 
bandwidth allocated to the sending nodal member 30. Packets are sent at the measurement 
bandwidth rate over the measurement period (5 minutes). Every measurement period, the 
number of packets sent is recalculated before the measurements packets sent. Measurement 
10 packets are sent until the vector is stopped or deleted. 

As the receiving nodal member 30 receives measurement packets, the nodal member 
pre-processes them into a unit of data referred to as an Atomic Packet. The Atomic Packet 
stores information such as the packet ID, Vector ID, sending nodal member ID, transmit 
timestamp, receive timestamp, original TTL value and received TTL value, as well as the 
1 5 status of the various regions such as the IP header, UDP/TCP/Other header, payload and 
CQOS header. 

~ Once the measurement period is over, which is indicated by a message from the 

sending nodal member 30, the receiving nodal member 30 processes the Atomic Packets via 
its algorithms (as described above). Once completed, this information is stored for up to 36+ 
20 hours. The information is then sent to the service daemon 60 via the CQOS Protocol. If the 
service daemon 60 does not receive the result packet until some time later than expected, or if 
the service daemon receives a subsequent results packet, the service daemon polls the nodal 
member 30 for the results. The service daemon 60 can poll the nodal member 30 for data that 
was computed/measured 36+ hours in the past. 

25 By computing Atomic Packets and then reducing that information down to a small 

amount of information (the core metrics), the Internet metric system 10 allows for a very 
scalable system that is highly distributed. In addition, since the results data is constant in size 
regardless of the number of measurement packets sent, the system is far more efficient at 
storing data and reporting data. 

30 Although the invention has been described in language specific to computer structural 

features, methodological acts, and by computer readable media, it is to be understood that the 
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invention defined in the appended claims is not necessarily limited to the specific structures, 
acts, or media described. Therefore, the specific structural features, acts and mediums are 
disclosed as exemplary embodiments implementing the claimed invention. 

Furthermore, the various embodiments described above are provided by way of 
illustration only and should not be construed to limit the invention. Those skilled in the art 
will readily recognize various modifications and changes that may be made to the present 
invention without following the example embodiments and applications illustrated and 
described herein, and without departing from the true spirit and scope of the present 
invention, which is set forth in the following claims. 



